Skip to content

implementation of table pragmas generator#212

Open
earthquakeonmars wants to merge 1 commit into
okbob:masterfrom
earthquakeonmars:pragma_generator
Open

implementation of table pragmas generator#212
earthquakeonmars wants to merge 1 commit into
okbob:masterfrom
earthquakeonmars:pragma_generator

Conversation

@earthquakeonmars

Copy link
Copy Markdown
Contributor

Table pragmas generator for CREATE TEMP TABLE ... AS statements

Motivation

When a PL/pgSQL function creates a temporary table with CREATE TEMP TABLE ... AS SELECT
and then references that table in the following statements, plpgsql_check reports
error:42P01 ... relation ... does not exist - the checker doesn't execute the body, so
the table is never created. The existing workaround - a manually written
plpgsql_check_pragma('table: t(col1 type1, ...)') - works, but the column list must be
duplicated by hand and it gets out of sync easily when the query changes.

This PR makes these pragmas generated automatically.

What is new

Two additions to the SQL API:

  • a new function

    plpgsql_check_generate_table_pragmas(funcoid regprocedure,
                                         relid regclass DEFAULT 0,
                                         fatal_errors boolean DEFAULT true)
    RETURNS SETOF text
    

    It scans the function's body and returns one table: name(col type, ...) pragma string
    for every CREATE TEMP TABLE ... AS SELECT|VALUES|TABLE statement there. Column names
    and types are derived from planning of the inner query (the query is planned, never
    executed). relid has the same meaning as in plpgsql_check_function - it is needed
    for trigger functions.

  • a new option pragmas text[] DEFAULT NULL for all four variants of
    plpgsql_check_function and plpgsql_check_function_tb. Any pragma string is accepted
    there and applied before the check is started.

The whole workflow is one line:

select * from plpgsql_check_function('fx()',
    pragmas => array(select plpgsql_check_generate_table_pragmas('fx()')));

Example:

create or replace function fx()
returns void as $$
begin
  create temp table xx as select a, b from tab;
  insert into xx values (10, 'hello');
end;
$$ language plpgsql;

select * from plpgsql_check_generate_table_pragmas('fx()');
-- table: xx(a integer, b text)

The returned text can be edited by the user before it is passed to the pragmas option.
The README got a new section "Table pragmas generator" (plus the pragmas option is
documented in the arguments list, and the "Temporary tables" limits section points to the
generator now).

Versioning and compatibility

This is an SQL API change, so I bumped the extension version to 2.10 in all the places
(following the pattern of the 2.7 -> 2.8 bump): the script is renamed to
plpgsql_check--2.10.sql, default_version in the control file, EXPECTED_EXTVERSION,
PG_MODULE_MAGIC_EXT (2.10.0), meson project version, META.json, DATA in Makefile, and
all seven spec files.

Please confirm the version number choice (2.10.0) - release numbering is yours, and I
don't want your "prepare for X.Y.Z" process to trip over an already-done bump. I can
renumber the whole set easily if you prefer something else.

Backward compatibility: the new parameters are appended at the end of the signatures
with DEFAULT NULL, so all existing calls keep working unchanged. Since the project
doesn't support ALTER EXTENSION ... UPDATE, users have to drop and create the extension
after upgrading the package - just a reminder, nothing new here.

Implementation notes (things worth a closer look in review)

  • The generator runs inside plpgsql_check_function_internal (new flag
    generate_pragmas in plpgsql_check_info), so it reuses SPI, the fake fcinfo/estate,
    the plpgsql compilation and - most importantly - the check's always-rolled-back
    subtransaction. Result: zero catalogue artifacts and idempotence (pinned by a
    regress test that counts pg_class rows before/after repeated calls).
  • One behaviour-preserving refactoring: the detection of inline pragmas
    (PERFORM plpgsql_check_pragma(...) and the PERFORM 'PRAGMA: ...' short form) is
    moved from the static ExprGetQuery into the exported
    plpgsql_check_apply_inline_pragmas, so the generator can reuse it. Please review this
    hunk primarily for equivalence with the old code.
  • Two more small exports: plpgsql_check_expr_prepare_plan (a thin wrapper around the
    static prepare_plan without plan checks) and plpgsql_check_put_text_line (a wrapper
    around the static put_text_line).
  • The 42P16 check ("cannot create temporary relation in non-temporary schema") is
    detected from the raw parse tree by the generator itself, intentionally not via
    RangeVarAdjustRelationPersistence - that routine could create the session's temporary
    namespace as a side effect.
  • fatal_errors => false semantics: a failed statement is skipped with a WARNING that
    keeps the original SQLSTATE and message, and the scanning continues. With the default
    fatal_errors => true the first failure is raised.
  • In generator mode errors are re-thrown as real errors instead of being converted to
    result rows - the output stays a clean list of pragmas, safe to feed into the pragmas
    option programmatically.

Behaviour details

  • The inner query is not limited to a simple FROM - joins, subqueries, CTE and recursive
    CTE queries work.
  • Statements are processed in the order of appearance in the body (nested blocks, loops,
    IF branches and exception handlers are scanned too); pragmas are returned positionally,
    without deduplication by name.
  • Every found table is registered immediately (through the existing table pragma
    mechanism), so chains "temp table built from an earlier temp table" resolve correctly.
  • Explicitly written PERFORM plpgsql_check_pragma('table: ...') calls in the body are
    respected - a manually declared table is visible when the following CTAS statements are
    planned.
  • Types are rendered with typmods (character varying(10), numeric(12,2)), identifiers
    are quoted when necessary, unknown is replaced by text (like CREATE TABLE AS
    does), explicit column lists t(c1, c2) AS ... override the names positionally.

What the function does not handle

1. Intentionally skipped, but still schema-checked (42P16)

All forms based on the CreateStmt parse node:

  • column definition list - CREATE TEMP TABLE t (id int, name text);
  • CREATE TEMP TABLE t (LIKE src INCLUDING ALL);
  • CREATE TEMP TABLE t OF my_type;
  • the same forms without the TEMP keyword, via the schema - CREATE TABLE pg_temp.t (id int);
    (recognized as temporary, but no pragma is generated - it is the list form)
  • list-form variations: INHERITS (parent), PARTITION OF parent FOR VALUES ..., any
    WITH (...) / USING method - all of these are the same CreateStmt node, the
    generator doesn't distinguish them.

For all of them the target schema is still validated: CREATE TEMP TABLE public.t (...)
raises 42P16 (or a WARNING with fatal_errors => false), but no pragma string is
returned. Because these forms produce the CreateStmt parse node (a different shape than
CreateTableAsStmt), the code for them will be done a bit later, as a follow-up.

2. Not visible to the generator at all

  • dynamic SQL - EXECUTE 'CREATE TEMP TABLE ...', including EXECUTE format(...) and
    query text from a variable. The workaround is a manual
    plpgsql_check_pragma('table: ...') next to the EXECUTE - the generator sees it and
    takes it into account for the following CTAS statements.
  • CREATE TEMP TABLE t AS EXECUTE prep_stmt - CTAS over a prepared statement; the
    structure cannot be derived statically, it is silently skipped.
  • tables created in called routines - PERFORM helper() / CALL proc() where the callee
    creates the temporary table. Only the body of the checked function is scanned, the
    generator doesn't descend into callees.
  • SELECT ... INTO TEMP t - in plain SQL this creates a table, but inside plpgsql this
    syntax is not possible (INTO is taken by plpgsql itself), and via EXECUTE it is
    dynamic SQL again.

3. Pragma string is returned, but cannot be applied

Corner cases where the output is honest, but registering/using the pragma ends with the
existing pragma parser's WARNING:

  • zero-column table - CREATE TEMP TABLE t AS SELECT FROM src; returns table: t(),
    but the pragma mechanism doesn't parse an empty column list.
  • duplicated unnamed columns - SELECT a+1, a+1 returns
    table: t("?column?" integer, "?column?" integer); note that such CTAS would fail at
    runtime too (column "?column?" specified more than once), so the function is broken
    by itself.

4. Lost details (precision limits, not failures)

  • column COLLATE is not carried into the pragma - the table: pragma parser doesn't
    support it, so a table registered by the pragma gets default collations.
  • related: CREATE TEMP SEQUENCE - the sequence: pragma exists in plpgsql_check, but
    the generator produces only table: pragmas; temporary sequences still have to be
    declared manually.

Testing and CI

  • New regress file plpgsql_check_pragma_generator (a single expected file for all
    supported versions) covering: all CTAS forms (AS SELECT / AS VALUES / AS TABLE,
    WITH NO DATA, IF NOT EXISTS, ON COMMIT variants, explicit column lists), name and
    type rendering (quoting, ?column?, unknown -> text, arrays/composites/domains/
    typmods, resjunk ORDER BY columns), body traversal (IF branches, loops, nested blocks,
    exception handlers, unreachable code), positional duplicates, table-from-table chains,
    shadowing of a real table, pg_temp qualification, skipped forms, plpgsql variables
    inside the query, overloaded functions and OUT parameters, trigger functions with
    relid, error cases (42P01/42P16/too many column names, fatal_errors => false
    branches with pinned SQLSTATEs), idempotence/no-artifacts, and the pragmas option
    end-to-end (incl. NULL/empty array, NULL elements, broken pragma string).
  • The test is added to REGRESS in Makefile and to the meson tests list (side note:
    the meson list historically doesn't run the profiler test; the new test is included).
    expected/README is updated.
  • Verified locally on PostgreSQL 18.4 - the full installcheck is green. PG 14-17 is
    covered only by CI here
    - if the output drifts on some version, I'm ready to add
    alternative expected files. The only version-sensitive assumption in the code is the
    CreateTableAsStmt.objtype field name (it is already objtype in PG14).
  • The meson changes were not exercised locally - tell me how you'd like them verified.
  • cppcheck and pgindent were run over the changed code; no new typedefs were introduced.

Deferred work

  • Generating pragmas for the CreateStmt-based forms of temporary tables (column
    definition list, LIKE, OF type_name) - planned as a follow-up.

…ble_pragmas

The new function plpgsql_check_generate_table_pragmas(funcoid, relid, fatal_errors)
scans the body of a plpgsql function, and for every CREATE TEMP TABLE AS statement
there generates the table pragma string. The names and types of columns are derived
from planning of the inner query, so the pragmas don't need to be written (and
maintained) manually.

Generated pragmas can be passed to the new "pragmas" option (text array) of
plpgsql_check_function and plpgsql_check_function_tb. These pragmas are applied
before the start of static analysis:

    create function fx()
    returns void as $$
    begin
      create temp table xx as select a, b from tab;
      insert into xx values (10, 'hello');
    end;
    $$ language plpgsql;

    select * from plpgsql_check_generate_table_pragmas('fx()');
    -- table: xx(a integer, b text)

    -- now the check doesn't fail on missing temporary table
    select * from plpgsql_check_function('fx()',
        pragmas => array(select plpgsql_check_generate_table_pragmas('fx()')));

Only statically written CREATE TEMP TABLE AS SELECT|VALUES|TABLE commands are
processed, dynamic SQL is ignored. Found temporary tables are registered by the
table pragma mechanism immediately, so the following statements of the function's
body can use them. The check of the target schema (42P16) is done for all forms
of temporary tables. When the option fatal_errors is false, then failed statements
are skipped with a warning only. The generator has no side effects - all created
objects are removed by rollback of the check's subtransaction.

This is SQL API change - the extension version is bumped to 2.10.
@okbob

okbob commented Jul 13, 2026

Copy link
Copy Markdown
Owner

it looks well - please change prefix from plpgsql_check_ to plch_ . I use it for new code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants