Basic question about structuring SQL
I'm an experienced programmer but really new to SQL, and I'm wanting a pointer to "the SQL way" to structure/organise chunks of code. A while back, I wrote a chunk of SQL to Do Something Useful. I put it in a file (do-something-useful.sql). And, to protect against getting into a weird state, I wrapped the code in my file with BEGIN; UPDATE DELETE... COMMIT; With the idea that I can do psql my_database \i do-something-useful.sql And be sure that either my task will be have been completed, or nothing with have changed. NOW, I want to do the same for a BIGGER task. BUT I realise that if I create bigger-task.sql like this... BEGIN; <> \i do-something-useful.sql <> COMMIT; ...the COMMIT inside "do-something-useful.sql" closes the transaction started in "bigger-task.sql" So I can get some things committed even if other things (in tidy-up) fail. So how SHOULD I tackle this? PostgreSQL does not do nested transactions (right?) So how should I structure my chunks of SQL so that I can have "safe" (all-or-nothing) blocks, AND use them from within one another? Robert
Writing WAL files
I am moving a database from PSQL 9 (!) on CentOS 6 to PSQL 12 on CentOS 7 I have a pair of servers -- one live, one standby. The live server defines an archive_command as "rsync" to shift WAL files to the standby server, The standby server uses "pg_standby" to monitor and process incoming WAL files. I believe this is all very vanilla, and indeed changes made in the live database are duly shipped to the standby. BUT... One of the things I like about the old PGSQL 9 setup is that it generates and ships a WAL file every few minutes, even if nothing has happened in the database. I find it re-assuring to be able to see WAL files arriving and being processed without problem even when the live system was idle. But I cannot get PGSQL 12 to do this. It only writes (and thus ships) WAL files when something happens in the database. If the database is idle, it simply does not write any WAL files. I thought I would get WAL files written from an idle database if, in postgresql.conf, I set "archive_timeout" to 120. And I've tried setting "checkpoint_timeout" to 90s, But to no avail. No WAL files are written unless the database changes. So what am I missing? How CAN I get postgresql 12 to write "unnecessary" WAL files every couple of minutes? Robert. -- Robert Inder,0131 229 1052 / 07808 492 213 Interactive Information Ltd, 3, Lauriston Gardens, Edinburgh EH3 9HH Registered in Scotland, Company no. SC 150689 Interactions speak louder than words
Re: Writing WAL files
On Sun, 4 Oct 2020 at 18:01, Adrian Klaver wrote: > On 10/4/20 9:54 AM, Robert Inder wrote: > > I am moving a database from PSQL 9 (!) on CentOS 6 to PSQL 12 on CentOS 7 > > It would help to know what the x in 9.x is? Before version 10 of > Postgres, the second number denoted a major version. > 9.4. Moving to 12.4. > > > > > I have a pair of servers -- one live, one standby. > > The live server defines an archive_command as "rsync" to shift WAL > > files to the standby server, > > The standby server uses "pg_standby" to monitor and process incoming WAL > > files. > > I believe this is all very vanilla, and indeed changes made in the live > > database are duly shipped to the standby. > > > > BUT... > > > > One of the things I like about the old PGSQL 9 setup is that it > > generates and ships a WAL file every few minutes, even if nothing has > > happened in the database. > > I find it re-assuring to be able to see WAL files arriving and being > > processed without problem even when the live system was idle. > > > > But I cannot get PGSQL 12 to do this. It only writes (and thus ships) > > WAL files when something happens in the database. > > If the database is idle, it simply does not write any WAL files. > > > > I thought I would get WAL files written from an idle database if, in > > postgresql.conf, I set "archive_timeout" to 120. > > > > And I've tried setting "checkpoint_timeout" to 90s, > > > > But to no avail. No WAL files are written unless the database changes. > > > > So what am I missing? How CAN I get postgresql 12 to write > > "unnecessary" WAL files every couple of minutes? > > > > Robert. > > > > -- > > Robert Inder,0131 229 1052 / 07808 > > 492 213 > > Interactive Information Ltd, 3, Lauriston Gardens, Edinburgh EH3 9HH > > Registered in Scotland, Company no. SC 150689 > > Interactions speak louder > > than words > > > -- > Adrian Klaver > adrian.kla...@aklaver.com > > > -- Robert Inder,0131 229 1052 / 07808 492 213 Interactive Information Ltd, 3, Lauriston Gardens, Edinburgh EH3 9HH Registered in Scotland, Company no. SC 150689 Interactions speak louder than words
Re: Writing WAL files
On Sun, 4 Oct 2020 at 20:52, Alvaro Herrera wrote: > > This is on purpose; archiving WAL files that contain nothing is pure > wastage of good electrons. Seriously? Oh, holy > I suggest that in PG12 you can monitor the > "lag" of a standby server more directly by looking at columns write_lag, > flush_lag, replay_lag in the pg_stat_replication view. And are those things updated when there are no changes to the master database? If so, can anyone make the case that continually checking and updating them (how often?) wastes fewer electrons than shipping an empty file every few minutes? Or are they only measured when something is updated? If I upgrade/install/reconfigure/restart something, I want to know that I haven't broken the sync. Will looking at the replay_lag (where? master? standby?) tell me that the sync is still good? Or will they capture the last sync. operation, and so only tell me what I need to know if I do some kind of database operation? And if I have to do some kind of database operation, I may as well stick wiht the current arrangement, since that operation would force a WAL file transfer anyway... (You'll need to > change your configuration so that it uses streaming replication instead > of pg_standby and rsync, but that's far more convenient so it's a good > change anyway.) > Maybe, but it's forcing me to spend time understanding stuff that I really don't want to know about. Robert. -- Robert Inder,0131 229 1052 / 07808 492 213 Interactive Information Ltd, 3, Lauriston Gardens, Edinburgh EH3 9HH Registered in Scotland, Company no. SC 150689 Interactions speak louder than words
Re: Writing WAL files
On Mon, 5 Oct 2020 at 18:29, Michael Lewis wrote: > > If you setup a scripted process to update a single row with a timestamptz > on the source/primary every minute, then you have a very simple consistent > change and also a way to check on the replica what is current time vs > last_scripted_update_time if you will and know the approx lag. It would > seem like a simple albeit hacky solution to you wanting a file every X > minutes regardless of server activity. > I'd thought of going half way to that, and just have a cron job for psql -c 'CHECKPOINT' which doesn't give me the quantitative indication I'd get from updating a timestamp, but doesn't commit me to know about any particular database, either. The thing that I find most disconcerting is that the documentation for Version 12 says checkpoint_timeout is "Maximum time between automatic WAL checkpoints" But the change Adrian Klaverd highlighted suggests that this is deliberately no longer the case, and I am left wondering what it does, in fact do/mean now. By the by, top-posting (reply above all quoted text) is discouraged on > these groups. > Indeed. Once upon a time, my signature use to be along the lines of So what? It's easier for me, so I'll do it! > > What's wrong with top posting? > It makes it hard to see comments in context. Robert. -- Robert Inder,0131 229 1052 / 07808 492 213 Interactive Information Ltd, 3, Lauriston Gardens, Edinburgh EH3 9HH Registered in Scotland, Company no. SC 150689 Interactions speak louder than words
