I believe the problem in the unoptomized case is in expand_expr_real_1, where we have:
case NON_LVALUE_EXPR: case NOP_EXPR: case CONVERT_EXPR: This is a conversion between what, two pointer types? If so, I think there should be a special case here to check for converting between two pointer types and call convert_memory_address if so. Also, I think convert_memory_address ought to have a gcc_assert (GET_MODE (x) == to_mode); in the #ifndef case.