Postgresql error : PANIC: could not locate a valid checkpoint record

[Mahendrakar, Prabhakar - Dell Team]

Hello,

Good Morning !

We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate 
a valid checkpoint record) and Postgres service fails to come up.

LOG:  starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 
4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG:  listening on IPv4 address "127.0.0.1", port 9003
LOG:  listening on IPv4 address "10.xx.xx.x..", port 9003
LOG:  listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG:  database system was shut down at 2022-06-09 10:19:24 CEST
LOG:  invalid primary checkpoint record
PANIC:  could not locate a valid checkpoint record
LOG:  startup process (PID 8773) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure
LOG:  database system is shut down

This issue is seen in both Windows and Linux OS platforms.

To Brief on the Scenario: Our product say example with Version A1 uses Postgres 
10 and in the latest version of our product (Say A2) we upgraded the Postgres 
to 13.
   When we try to upgrade our Product 
through InstallAnyWhere from A1 to A2, Postgres service fails with above 
mentioned error.

 Could you please suggest the probable cause of the issue. Let us know if you 
require any more information.

Thanks,
Prabhakar



Internal Use - Confidential

Re: Postgresql error : PANIC: could not locate a valid checkpoint record

[Mateusz Henicz]

Hi,
Have you done pg_upgrade post Postgres 13 installation?

https://www.postgresql.org/docs/13/pgupgrade.html

Cheers,
Mateusz

pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <
prabhakar.mahendr...@dellteam.com> napisał(a):

> Hello,
>
>
>
> Good Morning !
>
>
>
> We are facing *checkpoint* related issues from PostGreSQL 13.4 ( could
> not locate a valid checkpoint record) and Postgres service fails to come up.
>
>
> 
>
> *LOG:  starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc
> (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit*
>
> *LOG:  listening on IPv4 address "127.0.0.1", port 9003*
>
> *LOG:  listening on IPv4 address "10.xx.xx.x..", port 9003*
>
> *LOG:  listening on Unix socket "/tmp/.s.PGSQL.9003"*
>
> *LOG:  database system was shut down at 2022-06-09 10:19:24 CEST*
>
> *LOG:  invalid primary checkpoint record*
>
> *PANIC:  could not locate a valid checkpoint record*
>
> *LOG:  startup process (PID 8773) was terminated by signal 6: Aborted*
>
> *LOG:  aborting startup due to startup process failure*
>
> *LOG:  database system is shut down*
>
>
>
> This issue is seen in both Windows and Linux OS platforms.
>
>
>
> To Brief on the Scenario: Our product say example with Version A1 uses
> Postgres 10 and in the latest version of our product (Say A2) we upgraded
> the Postgres to 13.
>
>When we try to upgrade our
> Product through InstallAnyWhere from A1 to A2, Postgres service fails with
> above mentioned error.
>
>
>
>  Could you please suggest the probable cause of the issue. Let us know if
> you require any more information.
>
>
>
> Thanks,
>
> Prabhakar
>
>
>
> Internal Use - Confidential
>

Re: Postgresql error : PANIC: could not locate a valid checkpoint record

[Mateusz Henicz]

Assuming you have shut down your postgres properly before upgrading, it
should be safe for you to run pg_resetwal.
https://www.postgresql.org/docs/current/app-pgresetwal.html
It should help in this case.

pt., 17 cze 2022 o 12:03 Mahendrakar, Prabhakar - Dell Team <
prabhakar.mahendr...@dellteam.com> napisał(a):

> Yes, We are using the pg_upgrade utility of Postgres.
>
>
>
> *From:* Mateusz Henicz 
> *Sent:* Friday, June 17, 2022 3:31 PM
> *To:* Mahendrakar, Prabhakar - Dell Team
> *Cc:* pgsql-general@lists.postgresql.org
> *Subject:* Re: Postgresql error : PANIC: could not locate a valid
> checkpoint record
>
>
>
> [EXTERNAL EMAIL]
>
> Hi,
>
> Have you done pg_upgrade post Postgres 13 installation?
>
>
> https://www.postgresql.org/docs/13/pgupgrade.html [postgresql.org]
> 
>
>
>
> Cheers,
> Mateusz
>
>
>
> pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team <
> prabhakar.mahendr...@dellteam.com> napisał(a):
>
> Hello,
>
>
>
> Good Morning !
>
>
>
> We are facing *checkpoint* related issues from PostGreSQL 13.4 ( could
> not locate a valid checkpoint record) and Postgres service fails to come up.
>
>
> 
>
> *LOG:  starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc
> (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit*
>
> *LOG:  listening on IPv4 address "127.0.0.1", port 9003*
>
> *LOG:  listening on IPv4 address "10.xx.xx.x..", port 9003*
>
> *LOG:  listening on Unix socket "/tmp/.s.PGSQL.9003"*
>
> *LOG:  database system was shut down at 2022-06-09 10:19:24 CEST*
>
> *LOG:  invalid primary checkpoint record*
>
> *PANIC:  could not locate a valid checkpoint record*
>
> *LOG:  startup process (PID 8773) was terminated by signal 6: Aborted*
>
> *LOG:  aborting startup due to startup process failure*
>
> *LOG:  database system is shut down*
>
>
>
> This issue is seen in both Windows and Linux OS platforms.
>
>
>
> To Brief on the Scenario: Our product say example with Version A1 uses
> Postgres 10 and in the latest version of our product (Say A2) we upgraded
> the Postgres to 13.
>
>When we try to upgrade our
> Product through InstallAnyWhere from A1 to A2, Postgres service fails with
> above mentioned error.
>
>
>
>  Could you please suggest the probable cause of the issue. Let us know if
> you require any more information.
>
>
>
> Thanks,
>
> Prabhakar
>
>
>
>
>
> Internal Use - Confidential
>
>
>
> Internal Use - Confidential
>
>

Re: Postgresql error : PANIC: could not locate a valid checkpoint record

[Laurenz Albe]

On Fri, 2022-06-17 at 06:35 +, Mahendrakar, Prabhakar - Dell Team wrote:
> We are facing checkpoint related issues from PostGreSQL 13.4 ( could not 
> locate a valid checkpoint record) and Postgres service fails to come up.
> 
> LOG:  starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 
> 4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
> LOG:  listening on IPv4 address "127.0.0.1", port 9003
> LOG:  listening on IPv4 address "10.xx.xx.x..", port 9003
> LOG:  listening on Unix socket "/tmp/.s.PGSQL.9003"
> LOG:  database system was shut down at 2022-06-09 10:19:24 CEST
> LOG:  invalid primary checkpoint record
> PANIC:  could not locate a valid checkpoint record
> LOG:  startup process (PID 8773) was terminated by signal 6: Aborted
> LOG:  aborting startup due to startup process failure
> LOG:  database system is shut down
>  
> This issue is seen in both Windows and Linux OS platforms. 
>  
> To Brief on the Scenario: Our product say example with Version A1 uses 
> Postgres 10 and in the
> latest version of our product (Say A2) we upgraded the Postgres to 13.
> When we try to upgrade our Product through InstallAnyWhere from A1 to A2,
> Postgres service fails with above mentioned error.

You should not be using 13.4, but the latest minor release, at this point 13.7.

Anyway, the error message looks like somebody removed the contents of the 
"pg_wal"
directory.  How exactly did you upgrade PostgreSQL?

Yours,
Laurenz Albe
-- 
Cybertec | https://www.cybertec-postgresql.com

RE: Postgresql error : PANIC: could not locate a valid checkpoint record

[Mahendrakar, Prabhakar - Dell Team]

Yes, We are using the pg_upgrade utility of Postgres.

From: Mateusz Henicz 
Sent: Friday, June 17, 2022 3:31 PM
To: Mahendrakar, Prabhakar - Dell Team
Cc: pgsql-general@lists.postgresql.org
Subject: Re: Postgresql error : PANIC: could not locate a valid checkpoint 
record


[EXTERNAL EMAIL]
Hi,
Have you done pg_upgrade post Postgres 13 installation?

https://www.postgresql.org/docs/13/pgupgrade.html 
[postgresql.org]

Cheers,
Mateusz

pt., 17 cze 2022 o 11:20 Mahendrakar, Prabhakar - Dell Team 
mailto:prabhakar.mahendr...@dellteam.com>> 
napisał(a):
Hello,

Good Morning !

We are facing checkpoint related issues from PostGreSQL 13.4 ( could not locate 
a valid checkpoint record) and Postgres service fails to come up.

LOG:  starting PostgreSQL 13.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 
4.4.7 20120313 (Red Hat 4.4.7-16), 64-bit
LOG:  listening on IPv4 address "127.0.0.1", port 9003
LOG:  listening on IPv4 address "10.xx.xx.x..", port 9003
LOG:  listening on Unix socket "/tmp/.s.PGSQL.9003"
LOG:  database system was shut down at 2022-06-09 10:19:24 CEST
LOG:  invalid primary checkpoint record
PANIC:  could not locate a valid checkpoint record
LOG:  startup process (PID 8773) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure
LOG:  database system is shut down

This issue is seen in both Windows and Linux OS platforms.

To Brief on the Scenario: Our product say example with Version A1 uses Postgres 
10 and in the latest version of our product (Say A2) we upgraded the Postgres 
to 13.
   When we try to upgrade our Product 
through InstallAnyWhere from A1 to A2, Postgres service fails with above 
mentioned error.

 Could you please suggest the probable cause of the issue. Let us know if you 
require any more information.

Thanks,
Prabhakar



Internal Use - Confidential


Internal Use - Confidential

Re: Operators on ranges with a domain subtype do not implicitly cast operands of the domain's base type

[Daniel Popowich]

On Thu, Jun 16, 2022 at 4:26 PM Tom Lane  wrote:

> Daniel Popowich  writes:
> > -- domain with underlying type of integer (what constraints we might
> > -- place on the integer values are not germane to the issue so
> they're
> > -- left out).
> > create domain zzzint integer;
>
> > -- a range on our domain
> > create type zzzrange as range (subtype = zzzint);
>
> Why is this a good idea?
>
> ISTM the subtype of a range type shouldn't really be a domain.
>

In my case I have an integer representing a tax year.  Early in development
I found this "type" cropping up all over my schema and application logic.
Everywhere it occurred I was placing the same check constraints to make
sure it was an integer in our expected range of values,  I didn't want
years prior to a certain year, or years beyond one year into the future.
Didn't want people fat-fingering "2202", so:

CREATE DOMAIN taxyear INTEGER CONSTRAINT taxyear_range CHECK (value
BETWEEN 1980 AND  date_part('year', CURRENT_DATE) + 1);

This provides useful semantics throughout my schema and application code
(taxyear vs integer) and good data validation.  Really cleans up the code.
If the lower end of the range changes, I only have to change it in one
place, etc.

Meanwhile, there are entities in my data modeling that accept ranges of tax
years.  A questionnaire, for example, that might apply to a contiguous
range of years. Or a "study" of tax years, say, from 2018-2021.  I could
have implemented such models with begin/end years, but why?  The years are
always contiguous and I have the benefit of range operators, eg. given a
range I can now use `some_range @> some_taxyear` in a filter.  Very
powerful, clean, expressive.  Thus I created:

   CREATE TYPE tyrange AS RANGE (subtype = taxyear);

And so, here I am, getting user input of "2017" and expressions like

   SELECT * FROM questionnaire WHERE years @> 2017;

Are blowing up with:

ERROR:  operator does not exist: tyrange @> integer
LINE 1: select * from questionnaire where years @> 2017;

  ^
HINT:  No operator matches the given name and argument types. You might
need to add explicit type casts.

Forcing me to do explicit casts everywhere.  Given the underlying type of
taxyear is INTEGER and the operand is an INTEGER I'm finding this puzzling
why this is so difficult.

Hope that explains.
Daniel

Re: Operators on ranges with a domain subtype do not implicitly cast operands of the domain's base type

[Pavel Stehule]

pá 17. 6. 2022 v 17:26 odesílatel Daniel Popowich 
napsal:

> On Thu, Jun 16, 2022 at 4:26 PM Tom Lane  wrote:
>
>> Daniel Popowich  writes:
>> > -- domain with underlying type of integer (what constraints we might
>> > -- place on the integer values are not germane to the issue so
>> they're
>> > -- left out).
>> > create domain zzzint integer;
>>
>> > -- a range on our domain
>> > create type zzzrange as range (subtype = zzzint);
>>
>> Why is this a good idea?
>>
>> ISTM the subtype of a range type shouldn't really be a domain.
>>
>
> In my case I have an integer representing a tax year.  Early in
> development I found this "type" cropping up all over my schema and
> application logic.  Everywhere it occurred I was placing the same check
> constraints to make sure it was an integer in our expected range of
> values,  I didn't want years prior to a certain year, or years beyond one
> year into the future.  Didn't want people fat-fingering "2202", so:
>
> CREATE DOMAIN taxyear INTEGER CONSTRAINT taxyear_range CHECK (value
> BETWEEN 1980 AND  date_part('year', CURRENT_DATE) + 1);
>
> This provides useful semantics throughout my schema and application code
> (taxyear vs integer) and good data validation.  Really cleans up the code.
> If the lower end of the range changes, I only have to change it in one
> place, etc.
>
> Meanwhile, there are entities in my data modeling that accept ranges of
> tax years.  A questionnaire, for example, that might apply to a contiguous
> range of years. Or a "study" of tax years, say, from 2018-2021.  I could
> have implemented such models with begin/end years, but why?  The years are
> always contiguous and I have the benefit of range operators, eg. given a
> range I can now use `some_range @> some_taxyear` in a filter.  Very
> powerful, clean, expressive.  Thus I created:
>
>CREATE TYPE tyrange AS RANGE (subtype = taxyear);
>
> And so, here I am, getting user input of "2017" and expressions like
>
>SELECT * FROM questionnaire WHERE years @> 2017;
>
> Are blowing up with:
>
> ERROR:  operator does not exist: tyrange @> integer
> LINE 1: select * from questionnaire where years @> 2017;
>
> ^
> HINT:  No operator matches the given name and argument types. You might
> need to add explicit type casts.
>
> Forcing me to do explicit casts everywhere.  Given the underlying type of
> taxyear is INTEGER and the operand is an INTEGER I'm finding this puzzling
> why this is so difficult.
>

This feature has not yet been implemented, maybe. Or forgotten. The type
system (and internal implementation) is pretty complex because an
overloading, polymorphics types, domains are supported.

Probably the fix will not be too difficult - but can be hard to rethink all
consequences and dependencies.

Regards

Pavel



>
> Hope that explains.
> Daniel
>

Re: '{"x": 42, "y": null}'::jsonb != '{"x": 42}'::jsonb ... Really?

[Bryn Llewellyn]

> mmonc...@gmail.com  wrote:
> 
> yeah. I would expect for json or jsonb, two values, "a, b",  "a" is distinct 
> from "b" should give the same answer as "a::text is distinct from b::text".
> 
>> t...@sss.pgh.pa.us  wrote:
>> 
>>> b...@yugabyte.com  wrote:
>>> 
>>> I'm going to try to think like this: The number of possible spellings of 
>>> the names of keys in a JSON object is some flavor of infinite. So including 
>>> this in an object:
>>> 
>>> "k": null
>>> 
>>> really is saying something. It says that I do know about "k" and that yet I 
>>> have simply no information available about its value.
>> 
>> I'd read it as asserting that key "k" is meaningful for this object, but the 
>> correct value for that key is not known.
>> 
>> I have a hard time with your assertion that {"x": 42, "y": null} should be 
>> considered equivalent to {"x": 42}, because it would
>> render key-exists predicates useless. Either you have to say that key "y" is 
>> claimed to exist in both of these objects and indeed every object, or you 
>> have to make it fail if the key's value is null (so that it'd say "false" in 
>> both of these cases). Either of those options seems both weird and useless.
>> 
>>> The quirkiness that my first example showed can be seen differently from 
>>> how I saw it first. When I now consider this expression:
>>> 
>>> ('{"x": 42}'::jsonb)->>'y'
>>> 
>>> it seems that its evaluation should simply raise an exception. But you said:
>>> 
 This produces "key y not present in JSON" but someone decided that was too 
 unfriendly and so we instead produce SQL NULL.
>> 
>> Right. This is hard to justify from a purist semantic point of view, but 
>> having the operator throw an error in such cases would make it close to 
>> unusable on not-uniformly-structured data. And really the point of using 
>> JSON inside a SQL database is to cope with irregularly-structured data, so 
>> fuzziness seems like what we want.

Thank you very much for this, Tom. And thanks again to you, David, for your 
input. I hope that it's clear that the purpose of my questions is to discover 
what I'm missing—both w.r.t. actual semantics and w.r.t. the use cases that 
motivate PostgreSQL's functionality. Sorry if my questions (here and on other 
topics) might seem to challenge established wisdom and precedent.

Thank you very much for this, Tom. And thanks again to you, David, for your 
earlier input. I hope that it's clear that the purpose of my questions is to 
discover what I'm missing—both w.r.t. actual semantics and w.r.t. the use cases 
that motivate PostgreSQL's functionality. Sorry if, contrary to my intention, 
my questions (here and on other topics) might seem to challenge established 
wisdom and precedent.

I delayed my reply until I'd had time to think, to do some study, and (most 
importantly) to implement a complete, self-contained proof-of-concept to 
substantiate my conclusion. I'm ready, now, to report back.

Summary: I can meet my goal by using PG's native functionality appropriately.

So, w.r.t. this email’s subject, this:

  select strip_null_keys('{"x": 42, "y": null}'::jsonb) = '{"x": 42}'::jsonb;

returns "true".

The point at issue is whether the presence of « "some key": null » is different 
from the absence of "some key". And you (all) have argued that the two 
locutions for what I have wanted to see as one notion are indeed different. 
Well, yes, of course they are. This is tautologically true if you think only of 
the Unicode text of a JSON document that's so far waiting to be ingested by who 
knows what system. And you pointed out that, in PostgreSQL

  my_doc ? 'some key'

detects the difference. You've convinced me that some use cases will care about 
this—in other words, it's a distinction *with* a difference. But, in my use 
case, the two distinct locutions bring no semantic difference. But the actual 
distinction causes me a problem when I try to prove that this pair of 
transformations is idempotent:

  JSON → relational → JSON

But that problem is behind me now.

So... (and if you read this far at all) you can stop now unless you're 
interested to read further.

Before getting to my use case, I noted that Tom said this about the fact that 
using ->> to read a non-existent key returns a SQL null:

> [This] is hard to justify from a purist semantic point of view, but having 
> the operator throw an error in such cases would make it close to unusable on 
> not-uniformly-structured data.


It's easy to fix this, when the app requires strictness, by implementing a 
user-defined operator pair, say +>> and its partner. I did this (to respect 
what my use case needs) —and it passed all my tests. However, it turned out, 
for reasons that I explain below, that I didn't need it. The operator's 
implementation function reads the value with the appropriate native operator 
and only if it returns SQL null (or JSON null) does it do

ISBN (was: '{"x": 42, "y": null}'::jsonb != '{"x": 42}'::jsonb ... Really?)

[Peter J. Holzer]

On 2022-06-17 17:14:14 -0700, Bryn Llewellyn wrote:
> "isbn" — string
> values must be unique across the entire set of documents (in other words, it
> defines the unique business key); values must have this pattern:
> 
>   « ^[0-9]{3}-[0-9]{1}-[0-9]{2}-[0-9]{6}-[0-9]{1}$ »

Completely off-topic, but this regexp doesn't describe ISBNs. In ISBNs
the three middle subfields are all variable length. The first is a
language code (there are more than 10 languages in the world), the
second identifies the publisher (there are more than 100 publishers) and
the third the book. For example, "Mastering PostgreSQL 9.6" has the ISBN
978-1-78355-535-2.

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature

Re: ERROR: failed to find conversion function from key_vals_nn to record[]

[Peter J. Holzer]

On 2022-06-16 23:58:23 -0700, Bryn Llewellyn wrote:
> david.g.johns...@gmail.com wrote:
> 
> b...@yugabyte.com wrote:
> 
> Can anybody show me an implementation of a realistic use case that
> follows proper practice — like "every table must a primary key", "a
> foreign key must refer to a primary key", and "joins may be made only
> "on" columns one of which has a PK constraint and the other of which
> has a FK constraint" — where using a not nullable data type brings a
> problem that wouldn't occur if the column were defined with a nullable
> data type and an explicit "not null" constraint?
[...]
> -- "\d genres" shows "gk" with a "not null" constraint, whether I write it
> -- or not. And convention seems to say "don't clutter you code by writing
> it".
> create table genres(
>   gk  int   primary key,
>   gv  text  not null
>   );
>
> "Primary Key" is defined to be the application of both UNIQUE and NOT NULL
> constraints...
> 
> Yes, I know what "primary key" implies. I meant only to emphasize that the
> source column for what the "outer join" projects has a not null constraint,
> that it doesn't apply to the projection of that column, that this is perfectly
> understandable, and that this isn't a problem. Never mind.

But it would be a problem if there was an actual type which wouldn't
include NULL.

The NOT NULL attribute is an attribute of the column, not the type.
When you use the primary key (or any other column marked as NOT NULL)
the type of the result is just the type of that column, the NOT NULL is
dropped.

For example:

hjp=> \d genres
 Table "public.genres"
╔╤═╤═══╤══╤╗
║ Column │  Type   │ Collation │ Nullable │  Default   ║
╟┼─┼───┼──┼╢
║ id │ integer │   │ not null │ nextval('genres_id_seq'::regclass) ║
║ name   │ text│   │ not null │║
╚╧═╧═══╧══╧╝
Indexes:
"genres_pkey" PRIMARY KEY, btree (id)
Referenced by:
TABLE "books" CONSTRAINT "books_genre_fkey" FOREIGN KEY (genre) REFERENCES 
genres(id)

hjp=> \d books
 Table "public.books"
╔╤═╤═══╤══╤═══╗
║ Column │  Type   │ Collation │ Nullable │  Default  ║
╟┼─┼───┼──┼───╢
║ id │ integer │   │ not null │ nextval('books_id_seq'::regclass) ║
║ title  │ text│   │ not null │   ║
║ genre  │ integer │   │ not null │   ║
╚╧═╧═══╧══╧═══╝
Indexes:
"books_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
"books_genre_fkey" FOREIGN KEY (genre) REFERENCES genres(id)

hjp=> create table books_by_genre as
  select g.name as genre, b.title
  from genres g left join books b on g.id = b.genre;
SELECT 4
Time: 14.046 ms
hjp=> \d books_by_genre
  Table "public.books_by_genre"
╔╤══╤═══╤══╤═╗
║ Column │ Type │ Collation │ Nullable │ Default ║
╟┼──┼───┼──┼─╢
║ genre  │ text │   │  │ ║
║ title  │ text │   │  │ ║
╚╧══╧═══╧══╧═╝

As you can see, the type of the two columns is just "text" not "text not
null". And this is as is should be because the result indeed contains a
NULL value:

hjp=> select * from books_by_genre ;
╔═╤══╗
║genre│  title   ║
╟─┼──╢
║ Non-Fiction │ Mastering PostgreSQL 9.6 ║
║ SF  │ Idoru║
║ SF  │ Network Effect   ║
║ Romance │ (∅)  ║
╚═╧══╝
(4 rows)

Now, if title actually had a type which didn't include a null value,
this wouldn't be possible. Either the database would have to lie
(declare the column with a type but store a value which is impossible in
that type) or the query would have to fail or the database would have to
figure out a super type which does include that value. Neither
possibility seems very attractive to me (the first one is completely
unacceptable, the second one would be annoying, the third one might be
hard to implement).

hp

-- 
   _  | Peter J. Holzer| Story must make more sense than reality.
|_|_) ||
| |   | h...@hjp.at |-- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |   challenge!"


signature.asc
Description: PGP signature