Yaniv Dary has posted comments on this change. Change subject: history: updated etl process for scalablility ......................................................................
Patch Set 13: (3 comments) http://gerrit.ovirt.org/#/c/26952/13/tos_project/process/AggregationToDaily_3.4.item File tos_project/process/AggregationToDaily_3.4.item: Line 83: <elementParameter field="QUERYSTORE_TYPE" name="QUERYSTORE" value=""""/> Line 84: <elementParameter field="TECHNICAL" name="QUERYSTORE:REPOSITORY_QUERYSTORE_TYPE" value=""/> Line 85: <elementParameter field="TECHNICAL" name="QUERYSTORE:QUERYSTORE_TYPE" value="BUILT_IN"/> Line 86: <elementParameter field="GUESS_SCHEMA" name="GUESS_SCHEMA" value=""""/> Line 87: <elementParameter field="MEMO_SQL" name="QUERY" value=""SELECT 
 history_id, 
 history_datetime, 
 datacenter_id, 
 datacenter_status, 
 minutes_in_status, 
 datacenter_configuration_version
FROM datacenter_hourly_history
WHERE history_datetime >= (SELECT var_datetime
						 FROM history_configuration
						 WHERE var_name = 'lastDayAggr')
AND history_datetime < '"+TalendDate.addDate(context.lastDayAggr, 1,"dd")+"'
ORDER BY history_datetime,
 	 datacenter_id,
 	 datacenter_status""/> since you have lastDayAggr you should use this for both where clauses, not select again, since you can save db queries Line 88: <elementParameter field="LABEL" name="NOTE" value="This option only applies when deploying and running in the Talend Runtime"/> Line 89: <elementParameter field="CHECK" name="SPECIFY_DATASOURCE_ALIAS" value="false"/> Line 90: <elementParameter field="TEXT" name="DATASOURCE_ALIAS" value=""""/> Line 91: <elementParameter field="ENCODING_TYPE" name="ENCODING" value=""ISO-8859-15""/> Line 2931: <elementParameter field="QUERYSTORE_TYPE" name="QUERYSTORE" value=""""/> Line 2932: <elementParameter field="TECHNICAL" name="QUERYSTORE:REPOSITORY_QUERYSTORE_TYPE" value=""/> Line 2933: <elementParameter field="TECHNICAL" name="QUERYSTORE:QUERYSTORE_TYPE" value="BUILT_IN"/> Line 2934: <elementParameter field="GUESS_SCHEMA" name="GUESS_SCHEMA" value=""""/> Line 2935: <elementParameter field="MEMO_SQL" name="QUERY" value=""
SELECT DISTINCT 'lastDayAggr', var_datetime
FROM history_configuration
WHERE var_name = 'lastDayAggr'
""/> you are still selecting this. Line 2936: <elementParameter field="LABEL" name="NOTE" value="This option only applies when deploying and running in the Talend Runtime"/> Line 2937: <elementParameter field="CHECK" name="SPECIFY_DATASOURCE_ALIAS" value="false"/> Line 2938: <elementParameter field="TEXT" name="DATASOURCE_ALIAS" value=""""/> Line 2939: <elementParameter field="ENCODING_TYPE" name="ENCODING" value=""ISO-8859-15""/> http://gerrit.ovirt.org/#/c/26952/13/tos_project/process/HourlyTimeKeepingJob_3.4.item File tos_project/process/HourlyTimeKeepingJob_3.4.item: Line 391: <elementParameter field="QUERYSTORE_TYPE" name="QUERYSTORE" value=""""/> Line 392: <elementParameter field="TECHNICAL" name="QUERYSTORE:REPOSITORY_QUERYSTORE_TYPE" value=""/> Line 393: <elementParameter field="TECHNICAL" name="QUERYSTORE:QUERYSTORE_TYPE" value="BUILT_IN"/> Line 394: <elementParameter field="GUESS_SCHEMA" name="GUESS_SCHEMA" value=""""/> Line 395: <elementParameter field="MEMO_SQL" name="QUERY" value=""SELECT 'continueAgg', '1'
FROM history_configuration
WHERE var_name = 'lastHourAggr'
AND var_datetime < '"+TalendDate.addDate(RoutineHistoryETL.startOfHour(context.runTime), -1,"HH")+"'
LIMIT 1""/> Should not have limit 1. Will already return one row and this is not cross compatible Line 396: <elementParameter field="LABEL" name="NOTE" value="This option only applies when deploying and running in the Talend Runtime"/> Line 397: <elementParameter field="CHECK" name="SPECIFY_DATASOURCE_ALIAS" value="false"/> Line 398: <elementParameter field="TEXT" name="DATASOURCE_ALIAS" value=""""/> Line 399: <elementParameter field="ENCODING_TYPE" name="ENCODING" value=""ISO-8859-15""/> -- To view, visit http://gerrit.ovirt.org/26952 To unsubscribe, visit http://gerrit.ovirt.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6d7e6f8ad7238456a1e9cd05396cce217fa02a39 Gerrit-PatchSet: 13 Gerrit-Project: ovirt-dwh Gerrit-Branch: master Gerrit-Owner: Shirly Radco <sra...@redhat.com> Gerrit-Reviewer: Shirly Radco <sra...@redhat.com> Gerrit-Reviewer: Yaniv Dary <yd...@redhat.com> Gerrit-Reviewer: automat...@ovirt.org Gerrit-Reviewer: oVirt Jenkins CI Server Gerrit-HasComments: Yes _______________________________________________ Engine-patches mailing list Engine-patches@ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-patches
