[Engine-patches] Change in ovirt-dwh[master]: history: updated etl process for scalablility

[sradco]

Shirly Radco has posted comments on this change.

Change subject: history: updated etl process for scalablility
......................................................................


Patch Set 13:

(3 comments)

http://gerrit.ovirt.org/#/c/26952/13/tos_project/process/AggregationToDaily_3.4.item
File tos_project/process/AggregationToDaily_3.4.item:

Line 83:     <elementParameter field="QUERYSTORE_TYPE" name="QUERYSTORE" 
value="&quot;&quot;"/>
Line 84:     <elementParameter field="TECHNICAL" 
name="QUERYSTORE:REPOSITORY_QUERYSTORE_TYPE" value=""/>
Line 85:     <elementParameter field="TECHNICAL" 
name="QUERYSTORE:QUERYSTORE_TYPE" value="BUILT_IN"/>
Line 86:     <elementParameter field="GUESS_SCHEMA" name="GUESS_SCHEMA" 
value="&quot;&quot;"/>
Line 87:     <elementParameter field="MEMO_SQL" name="QUERY" 
value="&quot;SELECT &#xD;&#xA;  history_id, &#xD;&#xA;  history_datetime, 
&#xD;&#xA;  datacenter_id, &#xD;&#xA;  datacenter_status, &#xD;&#xA;  
minutes_in_status, &#xD;&#xA;  datacenter_configuration_version&#xD;&#xA;FROM 
datacenter_hourly_history&#xD;&#xA;WHERE history_datetime >= (SELECT 
var_datetime&#xD;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&#x9;   FROM 
history_configuration&#xD;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&#x9;   WHERE var_name 
= 'lastDayAggr')&#xA;AND history_datetime &lt; 
'&quot;+TalendDate.addDate(context.lastDayAggr, 
1,&quot;dd&quot;)+&quot;'&#xD;&#xA;ORDER BY history_datetime,&#xD;&#xA;      
&#x9; datacenter_id,&#xD;&#xA;      &#x9; datacenter_status&quot;"/>
> since you have lastDayAggr you should use this for both where clauses, not 
Done
Line 88:     <elementParameter field="LABEL" name="NOTE" value="This option 
only applies when deploying and running in the Talend Runtime"/>
Line 89:     <elementParameter field="CHECK" name="SPECIFY_DATASOURCE_ALIAS" 
value="false"/>
Line 90:     <elementParameter field="TEXT" name="DATASOURCE_ALIAS" 
value="&quot;&quot;"/>
Line 91:     <elementParameter field="ENCODING_TYPE" name="ENCODING" 
value="&quot;ISO-8859-15&quot;"/>


Line 2931:     <elementParameter field="QUERYSTORE_TYPE" name="QUERYSTORE" 
value="&quot;&quot;"/>
Line 2932:     <elementParameter field="TECHNICAL" 
name="QUERYSTORE:REPOSITORY_QUERYSTORE_TYPE" value=""/>
Line 2933:     <elementParameter field="TECHNICAL" 
name="QUERYSTORE:QUERYSTORE_TYPE" value="BUILT_IN"/>
Line 2934:     <elementParameter field="GUESS_SCHEMA" name="GUESS_SCHEMA" 
value="&quot;&quot;"/>
Line 2935:     <elementParameter field="MEMO_SQL" name="QUERY" 
value="&quot;&#xA;SELECT DISTINCT 'lastDayAggr', var_datetime&#xA;FROM 
history_configuration&#xA;WHERE var_name = 'lastDayAggr'&#xA;&quot;"/>
> you are still selecting this.
Done
Line 2936:     <elementParameter field="LABEL" name="NOTE" value="This option 
only applies when deploying and running in the Talend Runtime"/>
Line 2937:     <elementParameter field="CHECK" name="SPECIFY_DATASOURCE_ALIAS" 
value="false"/>
Line 2938:     <elementParameter field="TEXT" name="DATASOURCE_ALIAS" 
value="&quot;&quot;"/>
Line 2939:     <elementParameter field="ENCODING_TYPE" name="ENCODING" 
value="&quot;ISO-8859-15&quot;"/>


http://gerrit.ovirt.org/#/c/26952/13/tos_project/process/HourlyTimeKeepingJob_3.4.item
File tos_project/process/HourlyTimeKeepingJob_3.4.item:

Line 391:     <elementParameter field="QUERYSTORE_TYPE" name="QUERYSTORE" 
value="&quot;&quot;"/>
Line 392:     <elementParameter field="TECHNICAL" 
name="QUERYSTORE:REPOSITORY_QUERYSTORE_TYPE" value=""/>
Line 393:     <elementParameter field="TECHNICAL" 
name="QUERYSTORE:QUERYSTORE_TYPE" value="BUILT_IN"/>
Line 394:     <elementParameter field="GUESS_SCHEMA" name="GUESS_SCHEMA" 
value="&quot;&quot;"/>
Line 395:     <elementParameter field="MEMO_SQL" name="QUERY" 
value="&quot;SELECT 'continueAgg', '1'&#xA;FROM history_configuration&#xA;WHERE 
var_name = 'lastHourAggr'&#xA;AND var_datetime &lt; 
'&quot;+TalendDate.addDate(RoutineHistoryETL.startOfHour(context.runTime), 
-1,&quot;HH&quot;)+&quot;'&#xA;LIMIT 1&quot;"/>
> Should not have limit 1. Will already return one row and this is not cross 
Done
Line 396:     <elementParameter field="LABEL" name="NOTE" value="This option 
only applies when deploying and running in the Talend Runtime"/>
Line 397:     <elementParameter field="CHECK" name="SPECIFY_DATASOURCE_ALIAS" 
value="false"/>
Line 398:     <elementParameter field="TEXT" name="DATASOURCE_ALIAS" 
value="&quot;&quot;"/>
Line 399:     <elementParameter field="ENCODING_TYPE" name="ENCODING" 
value="&quot;ISO-8859-15&quot;"/>


-- 
To view, visit http://gerrit.ovirt.org/26952
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I6d7e6f8ad7238456a1e9cd05396cce217fa02a39
Gerrit-PatchSet: 13
Gerrit-Project: ovirt-dwh
Gerrit-Branch: master
Gerrit-Owner: Shirly Radco <sra...@redhat.com>
Gerrit-Reviewer: Shirly Radco <sra...@redhat.com>
Gerrit-Reviewer: Yaniv Dary <yd...@redhat.com>
Gerrit-Reviewer: automat...@ovirt.org
Gerrit-Reviewer: oVirt Jenkins CI Server
Gerrit-HasComments: Yes
_______________________________________________
Engine-patches mailing list
Engine-patches@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-patches