hmssql
Safe HaskellNone
LanguageHaskell2010

Hmssql.BulkCopy

Description

Bulk copy: many rows into one table in one request, by the protocol's bulk load (what bcp and .NET's SqlBulkCopy use). The rows go to the server as a stream of binary values, each in its column's own wire type, with no statement to compile and no parameters to bind, so it is the fastest way to load a table.

n <- bulkInsert conn "dbo.item" [(1 :: Int32, "one" :: Text, 9.99 :: Decimal), (2, "two", 0.5)]

bulkInsert takes any ToSqlRow value per row, asks the server for the table's columns, converts each value to its column's type on the client, and returns the number of rows the server inserted. bulkInsertWith adds the settings (BulkCopy: which columns, the server's options, batching); bulkInsertS and bulkInsertSWith take the rows from a Stream, with defaults or explicit settings. The rows of a list are streamed too, a packet at a time.

Rows. A positional row supplies one value per selected column, in the columns' order: under InsertableColumns the table's insertable columns in table order, so a derived ToSqlRow record must follow the table, where an INSERT would name its columns; under NamedColumns the columns named, in that order. A row with another number of values is an EncodeError naming the row. A row of NamedParams goes to the columns of its names instead, in any order, and must still name every selected column: the one leading @ a name may carry is dropped, so a and @a both name the column a, and @@a names a column called @a.

data NewItem = NewItem { itemId :: Int, itemName :: Text, itemPrice :: Decimal }
  deriving stock (Generic)
  deriving anyclass (ToSqlRow)

n <- bulkInsertWith (bulkCopy "dbo.item") { bulkColumns = NamedColumns ["id", "name", "price"] } conn items

What the server does with the rows. A bulk load skips the checks an INSERT makes unless asked for them: check constraints and foreign keys are not checked (bulkCheckConstraints), insert triggers don't fire (bulkFireTriggers), and a NULL for a column with a default gets the default (bulkKeepNulls). Identity columns are generated by the server unless explicitly selected (for example with bulkKeepIdentity). Automatic discovery excludes computed, rowversion and generated-always columns. A failing batch may be rolled back, while earlier batches remain committed; a trigger's RAISERROR can report an error without rolling back its rows. Use withTransaction for all-or-nothing loading.

Types. Each value is converted to its column's type as its ToSqlField instance sends it (encodeBulkCell has the rules). A value a column can't take (a Text for an int column, a string longer than the column, a character its code page lacks) is an EncodeError naming the row and column, thrown before the row is sent, where a plain INSERT can truncate or transcode it silently. A Double copied into a decimal or money column is rounded on the client from its exact binary value, half away from zero, as the server rounds it in a parameterized INSERT.

Permissions. An INSERT needs INSERT on the table. A bulk load also needs SELECT for the column discovery of InsertableColumns and NamedColumns (a SELECT TOP 0; DescribedColumns makes no such request), ALTER TABLE when the table has a check constraint or a trigger and bulkCheckConstraints or bulkFireTriggers is off (the server refuses the load with error 5304; with both on, INSERT suffices), and ALTER TABLE for bulkKeepIdentity. Measured on SQL Server 2025.

When an INSERT is the better tool. Bulk copy fills a table; for an application's ordinary inserts, execute or executeMany is usually right. A bulk load has no OUTPUT clause, so generated keys must be queried back; it checks and fires nothing unless asked, and then needs the permissions above; it is one table and whole rows, with no INSERT ... SELECT, MERGE or expression; the server answers once per batch, so an error in the first row surfaces after the whole batch is sent; it converts only within a type family where an INSERT converts a string to a number; and a load holds its locks to the end of its transaction. For a handful of rows, executeMany is nearly as fast and is one round trip where a load with column discovery is two.

On a MARS connection (connectMars) a load works but is a poor neighbor: with a result set open on the connection it fails with error 595 unless SET XACT_ABORT ON is in effect, inside a transaction any other request on the target table fails it with error 3970, and beside busy streams it runs several times slower. Large loads belong on a connection of their own.

Synopsis

Inserting rows

bulkInsert :: ToSqlRow r => Connection -> Text -> [r] -> IO Int64 Source #

Insert the rows into the table, one value per insertable column (InsertableColumns), and return how many rows the server inserted. bulkInsertWith (bulkCopy table).

bulkInsert conn "dbo.item" [(1 :: Int32, "one" :: Text), (2, "two")]

bulkInsertWith :: ToSqlRow r => BulkCopy -> Connection -> [r] -> IO Int64 Source #

bulkInsert with the settings.

bulkInsertWith (bulkCopy "dbo.item") { bulkColumns = NamedColumns ["id", "name"], bulkTableLock = True } conn rows

bulkInsertS :: ToSqlRow r => Connection -> Text -> Stream (Of r) IO () -> IO Int64 Source #

bulkInsert with rows from a stream and the default settings. Use bulkInsertSWith to choose columns, batching or other options. The streaming and interruption rules of bulkInsertSWith apply.

bulkInsertS conn "dbo.item" rows

bulkInsertSWith :: ToSqlRow r => BulkCopy -> Connection -> Stream (Of r) IO () -> IO Int64 Source #

bulkInsertWith with the rows from a stream, read as they are sent: a packet's worth at a time, so a stream of any length needs no more memory than that.

The stream runs while the request is being sent, with the connection locked: it must not use this connection (stream the rows from another one; on a MARS connection it may read a stream opened on this one before the load began, but not start a request). If it throws, the load is canceled and the exception is rethrown. Earlier batches stay committed. Cancellation normally leaves the connection usable, but the server can close it for an incomplete large bulk message. A row whose values can't be converted throws EncodeError the same way, naming the row (from 0) and the column. An asynchronous exception (a timeout expiring) while the rows are being sent breaks the connection instead, as an interrupted send does for any request.

With bulkBatchSize, each batch is sent and its response read before the next batch's rows are pulled from the stream. An empty stream sends nothing and returns 0.

The usual source is a query on another connection, which copies a table of any size a packet at a time:

src <- queryS source "SELECT id, name, price FROM dbo.item WHERE price > 1"
n <- bulkInsertS target "dbo.expensive_item" (src :: Stream (Of (Int, Text, Decimal)) IO ())

Settings

data BulkCopy Source #

How a bulk copy is done: the table, which of its columns, the server's bulk load options, and batching. bulkCopy has the defaults for a table.

Constructors

BulkCopy 

Fields

  • bulkTable :: !Text

    The table (or view) as SQL names it: dbo.item, [my schema].[t], #temp, db.dbo.t. Spliced into the statements as given, so quote it (escapeIdentifier style) if it comes from data.

  • bulkColumns :: !BulkColumns

    Which columns the rows hold, and in what order.

  • bulkKeepIdentity :: !Bool

    With InsertableColumns: send the identity column, with the rows' values, instead of letting the server generate it. (A column named in NamedColumns or given in DescribedColumns is sent as it is.) The load then needs ALTER TABLE permission on the table.

  • bulkKeepNulls :: !Bool

    KEEP_NULLS: insert a NULL as NULL where the column has a default; without it, the default is inserted instead.

  • bulkCheckConstraints :: !Bool

    CHECK_CONSTRAINTS: check the table's check and foreign key constraints. A bulk load skips them by default and leaves them marked as not trusted (is_not_trusted in sys.check_constraints and sys.foreign_keys), which keeps the optimizer from relying on them until ALTER TABLE ... WITH CHECK CHECK CONSTRAINT re-checks them; a load with this option keeps the trust. Without this option and bulkFireTriggers both, a table that has a check constraint or a trigger needs ALTER TABLE permission (error 5304).

  • bulkFireTriggers :: !Bool

    FIRE_TRIGGERS: run the table's insert triggers.

  • bulkTableLock :: !Bool

    TABLOCK: lock the whole table for the load, which allows minimal logging and is faster for a large load into a table nobody else is using.

  • bulkOrder :: ![BulkOrder]

    ORDER(...): the rows are sorted by these columns, as the clustered index is; the server then needn't sort them. Each batch must be in that order: a table with a clustered index refuses an out-of-order batch (error 4819, with or without bulkTableLock), and a heap ignores the hint.

  • bulkRowsPerBatch :: !(Maybe Int)

    ROWS_PER_BATCH: about how many rows the load has, for the query optimizer.

  • bulkKilobytesPerBatch :: !(Maybe Int)

    KILOBYTES_PER_BATCH: about how much data, likewise.

  • bulkBatchSize :: !(Maybe Int)

    Send the rows in batches of this many, each a bulk load of its own (its own INSERT BULK and round trip), committed on its own outside a transaction: a failure then loses only its batch's rows, and the server's memory for the load is bounded. Nothing (the default): every row in one batch, as one request.

Instances

Instances details
Show BulkCopy Source # 
Instance details

Defined in Hmssql.BulkCopy

bulkCopy :: Text -> BulkCopy Source #

The defaults for a table: every insertable column, no options, one batch.

data BulkColumns Source #

Which columns the rows hold.

Constructors

InsertableColumns

Every column of the table a plain INSERT without a column list would take, in table order: all but the identity column (unless bulkKeepIdentity), computed columns, rowversion and the columns the server generates. Asked of the server before the load, with a SELECT TOP 0 that needs SELECT permission on the table.

NamedColumns ![Text]

These columns, in this order; the rest get their defaults. Asked of the server before the load (their types), so a name that isn't a column is a SqlServerError.

DescribedColumns ![BulkColumn]

These columns, as described: no request to the server before the load. For a table loaded many times (describeBulkColumns gives the description once), for a login without SELECT on the table, or to declare the types the rows are sent in yourself: the server converts what arrives to the column's type (as an INSERT would, for a type it converts implicitly), so a varchar column can be loaded as nvarchar, say.

cols <- describeBulkColumns conn (bulkCopy "dbo.item")
let settings = (bulkCopy "dbo.item") { bulkColumns = DescribedColumns cols }
forM_ batches $ \rows -> bulkInsertWith settings conn rows

Instances

Instances details
Show BulkColumns Source # 
Instance details

Defined in Hmssql.BulkCopy

data BulkOrder Source #

A column of the ORDER hint.

Constructors

Ascending !Text 
Descending !Text 

Instances

Instances details
Show BulkOrder Source # 
Instance details

Defined in Hmssql.BulkCopy

Eq BulkOrder Source # 
Instance details

Defined in Hmssql.BulkCopy

The columns

data BulkColumn Source #

A column as the bulk load declares it: its name, the wire type the values are sent in (a length, precision and scale, or collation included; sqlTypeTypeInfo gives the type of a declaration such as varchar(20)), whether a NULL may be sent, and whether it is the table's identity column.

Instances

Instances details
Show BulkColumn Source # 
Instance details

Defined in Hmssql.BulkCopy

Eq BulkColumn Source # 
Instance details

Defined in Hmssql.BulkCopy

describeBulkColumns :: Connection -> BulkCopy -> IO [BulkColumn] Source #

The columns the rows of a bulk copy hold (bulkColumns): as given for DescribedColumns; for the others, asked of the server (SELECT TOP 0 ... FROM the table, one round trip). Each column keeps its real type (a varchar(50) column is a TIBigVarChar 50, with its collation), which is what encodeBulkCell validates a value against; the load sends it in a wire type (bulkWireType) the server transcodes from. Throws SqlServerError for a table or column that doesn't exist.

Low level

insertBulkStatement :: BulkCopy -> [BulkColumn] -> Text Source #

The INSERT BULK statement that announces the load: the table, each column with the type it is sent as, and the options.

bulkLoadPackets :: TdsContext -> Text -> [BulkColumn] -> [[RawBytes]] -> [ByteString] Source #

The packets of a bulk load message for these columns and rows (each row's values converted already, as encodeBulkCell gives them), as bulkInsertS sends them: COLMETADATA, a ROW per row, and the DONE, framed into packets of the context's size. For tests and benchmarks.

encodeBulkCell :: TdsContext -> BulkColumn -> EncodedParam -> Either String ParamValue Source #

Encode a value for a bulk load into this column: its bytes in the wire type the column is sent as (bulkWireType (bulkColumnType column)), or why the column can't take the value. The column's own type supplies the checks, and the wire type may differ: a varchar column's value is checked against the column's length and code page, then sent as UTF-16 in nvarchar(max) for the server to transcode.

A NULL is sent as NULL to a nullable column and refused for a NOT NULL one. A value's framing is checked as an ordinary parameter's is (valueProblem), so an empty numeric or sql_variant payload can't become a NULL by accident, and a value declared as a fixed char, nchar or binary is padded to its length before conversion, as an RPC parameter is. Where the column's type and its wire type agree, a value already in that type (an Int32 for an int column, a Text that fits an nvarchar column, or a matching RawParam) is sent as it is. A value converted to sql_variant keeps its declared base type, precision, scale, length and collation, with its bytes as they are. Other conversions decode the value and encode it again:

  • Integers, Bool, Decimal, Double, Float and Money to any of bit (nonzero is 1), the integer types (fractions truncate toward zero, except Money, which rounds half away from zero), decimal and numeric (rounded to the column's scale, half away from zero, a floating-point value from its exact binary value), float, real, money and smallmoney (rounded to four places); a value outside the column's range is refused. Nonfinite floating sources are refused for bit rather than converted to true.
  • Text (and String) to nchar, nvarchar, ntext (as UTF-16), char, varchar, text (validated against the column's UTF-8 or single-byte code page, then sent as UTF-16; a character the page lacks is refused), xml and sql_variant (as nvarchar, in the session's collation). A string longer than the column is refused. SqlXml goes to character and xml columns too. A character column gets its text as it is, without the server's XML canonicalization, and raw XML bytes in an encoding other than UTF-8 or a UTF-16 the client can identify are refused there.
  • ByteString to binary, varbinary, image and sql_variant; longer than the column, refused.
  • Day, TimeOfDay, LocalTime, UTCTime, DateTimeOffset and ZonedTime to date, time, datetime2, datetimeoffset (at the column's scale, rounded as the ToSqlField instances round), datetime and smalldatetime: a date at midnight, the date or time part of a date-time, a datetimeoffset's local time, a date-time with offset zero.
  • UUID to uniqueidentifier only; every value but SqlXml to sql_variant.

Anything else (a Text for an int column, a number for a string column, a CLR type) is refused: convert it in Haskell, or declare the column (DescribedColumns) with the type the value is sent as and let the server convert.

bulkWireType :: TypeInfo -> TypeInfo Source #

The wire type a column's values are sent as, which may differ from the column's real type (bulkColumnType). Character (char, varchar, text) and ntext columns are sent as nvarchar(max) (UTF-16) and transcoded by the server to the column's own collation on insert (as bcp -w does): correct whatever the collation, where a single-byte value would otherwise be read in the load's default collation, not the column's. image and xml are sent as varbinary(max); XML keeps its byte encoding for the server's parser. Everything else is sent as itself. encodeBulkCell still validates a value against the real type, so a character absent from the column's code page or a value too long is refused on the client, not silently transcoded or truncated by the server.