| Safe Haskell | None |
|---|---|
| Language | Haskell2010 |
Hmssql.BulkCopy
Description
Bulk copy: many rows into one table in one request, by the protocol's
bulk load (what bcp and .NET's SqlBulkCopy use). The rows go to the
server as a stream of binary values, each in its column's own wire type,
with no statement to compile and no parameters to bind, so it is the
fastest way to load a table.
n <- bulkInsert conn "dbo.item" [(1 :: Int32, "one" :: Text, 9.99 :: Decimal), (2, "two", 0.5)]
bulkInsert takes any ToSqlRow value per row, asks the server for
the table's columns, converts each value to its column's type on the
client, and returns the number of rows the server inserted.
bulkInsertWith adds the settings (BulkCopy: which columns, the
server's options, batching); bulkInsertS and bulkInsertSWith take
the rows from a Stream, with defaults or explicit settings. The rows of a list are streamed too, a packet at a time.
Rows. A positional row supplies one value per selected column, in
the columns' order: under InsertableColumns the table's insertable
columns in table order, so a derived ToSqlRow record must follow the
table, where an INSERT would name its columns; under NamedColumns the
columns named, in that order. A row with another number of values is an
EncodeError naming the row. A row of NamedParams
goes to the columns of its names instead, in any order, and must still
name every selected column: the one leading @ a name may carry is
dropped, so a and @a both name the column a, and @@a names a
column called @a.
data NewItem = NewItem { itemId :: Int, itemName :: Text, itemPrice :: Decimal }
deriving stock (Generic)
deriving anyclass (ToSqlRow)
n <- bulkInsertWith (bulkCopy "dbo.item") { bulkColumns = NamedColumns ["id", "name", "price"] } conn itemsWhat the server does with the rows. A bulk load skips the checks an
INSERT makes unless asked for them: check constraints and foreign
keys are not checked (bulkCheckConstraints), insert triggers don't
fire (bulkFireTriggers), and a NULL for a column with a default gets
the default (bulkKeepNulls). Identity columns are generated by the
server unless explicitly selected (for example with bulkKeepIdentity).
Automatic discovery excludes computed, rowversion and generated-always
columns. A failing batch may be rolled
back, while earlier batches remain committed; a trigger's RAISERROR
can report an error without rolling back its rows. Use
withTransaction for all-or-nothing loading.
Types. Each value is converted to its column's type as its
ToSqlField instance sends it (encodeBulkCell has the rules). A value
a column can't take (a Text for an int column, a string longer
than the column, a character its code page lacks) is an EncodeError
naming the row and column, thrown before the row is sent, where a
plain INSERT can truncate or transcode it silently. A Double
copied into a decimal or money column is rounded on the client
from its exact binary value, half away from zero, as the server rounds
it in a parameterized INSERT.
Permissions. An INSERT needs INSERT on the table. A bulk load
also needs SELECT for the column discovery of InsertableColumns and
NamedColumns (a SELECT TOP 0; DescribedColumns makes no such
request), ALTER TABLE when the table has a check constraint or a trigger
and bulkCheckConstraints or bulkFireTriggers is off (the server
refuses the load with error 5304; with both on, INSERT suffices), and
ALTER TABLE for bulkKeepIdentity. Measured on SQL Server 2025.
When an INSERT is the better tool. Bulk copy fills a table; for
an application's ordinary inserts, execute or
executeMany is usually right. A bulk load has no
OUTPUT clause, so generated keys must be queried back; it checks and
fires nothing unless asked, and then needs the permissions above; it is
one table and whole rows, with no INSERT ... SELECT, MERGE or
expression; the server answers once per batch, so an error in the first
row surfaces after the whole batch is sent; it converts only within a
type family where an INSERT converts a string to a number; and a load
holds its locks to the end of its transaction. For a handful of rows,
executeMany is nearly as fast and is one round trip where
a load with column discovery is two.
On a MARS connection (connectMars) a load works
but is a poor neighbor: with a result set open on the connection it
fails with error 595 unless SET XACT_ABORT ON is in effect, inside a
transaction any other request on the target table fails it with error
3970, and beside busy streams it runs several times slower. Large
loads belong on a connection of their own.
Synopsis
- bulkInsert :: ToSqlRow r => Connection -> Text -> [r] -> IO Int64
- bulkInsertWith :: ToSqlRow r => BulkCopy -> Connection -> [r] -> IO Int64
- bulkInsertS :: ToSqlRow r => Connection -> Text -> Stream (Of r) IO () -> IO Int64
- bulkInsertSWith :: ToSqlRow r => BulkCopy -> Connection -> Stream (Of r) IO () -> IO Int64
- data BulkCopy = BulkCopy {
- bulkTable :: !Text
- bulkColumns :: !BulkColumns
- bulkKeepIdentity :: !Bool
- bulkKeepNulls :: !Bool
- bulkCheckConstraints :: !Bool
- bulkFireTriggers :: !Bool
- bulkTableLock :: !Bool
- bulkOrder :: ![BulkOrder]
- bulkRowsPerBatch :: !(Maybe Int)
- bulkKilobytesPerBatch :: !(Maybe Int)
- bulkBatchSize :: !(Maybe Int)
- bulkCopy :: Text -> BulkCopy
- data BulkColumns
- data BulkOrder
- = Ascending !Text
- | Descending !Text
- data BulkColumn = BulkColumn {}
- describeBulkColumns :: Connection -> BulkCopy -> IO [BulkColumn]
- insertBulkStatement :: BulkCopy -> [BulkColumn] -> Text
- bulkLoadPackets :: TdsContext -> Text -> [BulkColumn] -> [[RawBytes]] -> [ByteString]
- encodeBulkCell :: TdsContext -> BulkColumn -> EncodedParam -> Either String ParamValue
- bulkWireType :: TypeInfo -> TypeInfo
Inserting rows
bulkInsert :: ToSqlRow r => Connection -> Text -> [r] -> IO Int64 Source #
Insert the rows into the table, one value per insertable column
(InsertableColumns), and return how many rows the server inserted.
.bulkInsertWith (bulkCopy table)
bulkInsert conn "dbo.item" [(1 :: Int32, "one" :: Text), (2, "two")]
bulkInsertWith :: ToSqlRow r => BulkCopy -> Connection -> [r] -> IO Int64 Source #
bulkInsert with the settings.
bulkInsertWith (bulkCopy "dbo.item") { bulkColumns = NamedColumns ["id", "name"], bulkTableLock = True } conn rowsbulkInsertS :: ToSqlRow r => Connection -> Text -> Stream (Of r) IO () -> IO Int64 Source #
bulkInsert with rows from a stream and the default settings.
Use bulkInsertSWith to choose columns, batching or other options.
The streaming and interruption rules of bulkInsertSWith apply.
bulkInsertS conn "dbo.item" rows
bulkInsertSWith :: ToSqlRow r => BulkCopy -> Connection -> Stream (Of r) IO () -> IO Int64 Source #
bulkInsertWith with the rows from a stream, read as they are sent:
a packet's worth at a time, so a stream of any length needs no more
memory than that.
The stream runs while the request is being sent, with the connection
locked: it must not use this connection (stream the rows from another
one; on a MARS connection it may read a stream opened on this one
before the load began, but not start a request). If it throws, the
load is canceled and the exception is rethrown. Earlier batches stay
committed. Cancellation normally leaves the connection usable, but the
server can close it for an incomplete large bulk message. A row whose
values can't be converted throws EncodeError the same way, naming
the row (from 0) and the column. An asynchronous exception (a
timeout expiring) while the rows are being sent
breaks the connection instead, as an interrupted send does for any
request.
With bulkBatchSize, each batch is sent and its response read before
the next batch's rows are pulled from the stream. An empty stream
sends nothing and returns 0.
The usual source is a query on another connection, which copies a table of any size a packet at a time:
src <- queryS source "SELECT id, name, price FROM dbo.item WHERE price > 1" n <- bulkInsertS target "dbo.expensive_item" (src :: Stream (Of (Int, Text, Decimal)) IO ())
Settings
How a bulk copy is done: the table, which of its columns, the
server's bulk load options, and batching. bulkCopy has the defaults
for a table.
Constructors
| BulkCopy | |
Fields
| |
bulkCopy :: Text -> BulkCopy Source #
The defaults for a table: every insertable column, no options, one batch.
data BulkColumns Source #
Which columns the rows hold.
Constructors
| InsertableColumns | Every column of the table a plain |
| NamedColumns ![Text] | These columns, in this order; the rest get their defaults. Asked
of the server before the load (their types), so a name that isn't
a column is a |
| DescribedColumns ![BulkColumn] | These columns, as described: no request to the server before the
load. For a table loaded many times ( cols <- describeBulkColumns conn (bulkCopy "dbo.item")
let settings = (bulkCopy "dbo.item") { bulkColumns = DescribedColumns cols }
forM_ batches $ \rows -> bulkInsertWith settings conn rows |
Instances
| Show BulkColumns Source # | |
Defined in Hmssql.BulkCopy Methods showsPrec :: Int -> BulkColumns -> ShowS # show :: BulkColumns -> String # showList :: [BulkColumns] -> ShowS # | |
The columns
data BulkColumn Source #
A column as the bulk load declares it: its name, the wire type the
values are sent in (a length, precision and scale, or collation
included; sqlTypeTypeInfo gives the type of a
declaration such as varchar(20)), whether a NULL may be sent, and
whether it is the table's identity column.
Constructors
| BulkColumn | |
Fields
| |
Instances
| Show BulkColumn Source # | |
Defined in Hmssql.BulkCopy Methods showsPrec :: Int -> BulkColumn -> ShowS # show :: BulkColumn -> String # showList :: [BulkColumn] -> ShowS # | |
| Eq BulkColumn Source # | |
Defined in Hmssql.BulkCopy | |
describeBulkColumns :: Connection -> BulkCopy -> IO [BulkColumn] Source #
The columns the rows of a bulk copy hold (bulkColumns): as given for
DescribedColumns; for the others, asked of the server (SELECT TOP 0
... FROM the table, one round trip). Each column keeps its real type
(a varchar(50) column is a , with its collation),
which is what TIBigVarChar 50encodeBulkCell validates a value against; the load sends
it in a wire type (bulkWireType) the server transcodes from. Throws
SqlServerError for a table or column that doesn't exist.
Low level
insertBulkStatement :: BulkCopy -> [BulkColumn] -> Text Source #
The INSERT BULK statement that announces the load: the table, each
column with the type it is sent as, and the options.
bulkLoadPackets :: TdsContext -> Text -> [BulkColumn] -> [[RawBytes]] -> [ByteString] Source #
The packets of a bulk load message for these columns and rows (each
row's values converted already, as encodeBulkCell gives them), as
bulkInsertS sends them: COLMETADATA, a ROW per row, and the DONE,
framed into packets of the context's size. For tests and benchmarks.
encodeBulkCell :: TdsContext -> BulkColumn -> EncodedParam -> Either String ParamValue Source #
Encode a value for a bulk load into this column: its bytes in the wire
type the column is sent as (), or
why the column can't take the value. The column's own type supplies the
checks, and the wire type may differ: a bulkWireType (bulkColumnType column)varchar column's value is checked
against the column's length and code page, then sent as UTF-16 in
nvarchar(max) for the server to transcode.
A NULL is sent as NULL to a nullable column and refused for a NOT NULL one.
A value's framing is checked as an ordinary parameter's is
(valueProblem), so an empty numeric or sql_variant
payload can't become a NULL by accident, and a value declared as a fixed
char, nchar or binary is padded to its length before conversion, as
an RPC parameter is. Where the column's type and its wire type agree, a
value already in that type (an Int32 for an int column, a Text that
fits an nvarchar column, or a matching RawParam) is sent as it is. A
value converted to sql_variant keeps its declared base type, precision,
scale, length and collation, with its bytes as they are. Other conversions
decode the value and encode it again:
- Integers,
Bool,Decimal,Double,FloatandMoneyto any ofbit(nonzero is 1), the integer types (fractions truncate toward zero, exceptMoney, which rounds half away from zero),decimalandnumeric(rounded to the column's scale, half away from zero, a floating-point value from its exact binary value),float,real,moneyandsmallmoney(rounded to four places); a value outside the column's range is refused. Nonfinite floating sources are refused forbitrather than converted to true. Text(andString) tonchar,nvarchar,ntext(as UTF-16),char,varchar,text(validated against the column's UTF-8 or single-byte code page, then sent as UTF-16; a character the page lacks is refused),xmlandsql_variant(asnvarchar, in the session's collation). A string longer than the column is refused.SqlXmlgoes to character andxmlcolumns too. A character column gets its text as it is, without the server's XML canonicalization, and raw XML bytes in an encoding other than UTF-8 or a UTF-16 the client can identify are refused there.ByteStringtobinary,varbinary,imageandsql_variant; longer than the column, refused.Day,TimeOfDay,LocalTime,UTCTime,DateTimeOffsetandZonedTimetodate,time,datetime2,datetimeoffset(at the column's scale, rounded as theToSqlFieldinstances round),datetimeandsmalldatetime: a date at midnight, the date or time part of a date-time, adatetimeoffset's local time, a date-time with offset zero.UUIDtouniqueidentifieronly; every value butSqlXmltosql_variant.
Anything else (a Text for an int column, a number for a string
column, a CLR type) is refused: convert it in Haskell, or declare the
column (DescribedColumns) with the type the value is sent as and let
the server convert.
bulkWireType :: TypeInfo -> TypeInfo Source #
The wire type a column's values are sent as, which may differ from the
column's real type (bulkColumnType). Character (char, varchar,
text) and ntext columns are sent as nvarchar(max) (UTF-16)
and transcoded by the server to the column's own collation on insert (as
bcp -w does): correct whatever the collation, where a single-byte value
would otherwise be read in the load's default collation, not the
column's. image and xml are sent as varbinary(max); XML keeps
its byte encoding for the server's parser. Everything else is sent
as itself. encodeBulkCell still validates a value against the real type,
so a character absent from the column's code page or a value too long is
refused on the client, not silently transcoded or truncated by the
server.