Safe Haskell	None
Language	Haskell2010

Hmssql.Tds.CodePage

Description

Decoding single-byte (char, varchar, text) data according to its collation.

A collation determines the code page of non-Unicode character data:

UTF-8 collations (the _UTF8 suffix, flag bit 26) use UTF-8.
SQL collations (SQL_*) have a non-zero sort ID, which determines the code page; the LCID does not (e.g. SQL_Latin1_General_CP1251_CI_AS has LCID 1033 but code page 1251).
Windows collations have sort ID 0; the LCID determines the code page.

The mappings and decode tables below were generated from SQL Server itself (SQL Server 2025), and are checked against it by the test suite:

SELECT name, COLLATIONPROPERTY(name, 'CodePage'), COLLATIONPROPERTY(name, 'LCID'),
       COLLATIONPROPERTY(name, 'SortId')
FROM sys.fn_helpcollations()

and, for each code page, the server's own conversion of every byte value to nvarchar. Bytes a code page leaves undefined decode the way the server (and Windows' MultiByteToWideChar) converts them: to the C1 control character with the same value (e.g. 0x81 in code page 1252 is U+0081), or, in code pages 874, 1253 and 1255, to a private-use character.

The double-byte code pages (932, 936, 949, 950) are not supported.

Synopsis

type CodePage = Word16
collationLcid :: Collation -> Word32
collationIsUtf8 :: Collation -> Bool
collationCodePage :: Collation -> Maybe CodePage
singleByteDecoder :: CodePage -> Maybe (ByteString -> Text)
collationTextDecoder :: Collation -> Either String (ByteString -> Either String Text)
collationTextEncoder :: Collation -> Either String (Text -> Either String ByteString)
singleByteEncoder :: CodePage -> Maybe (Text -> Either String ByteString)
data TextCodec
collationCodec :: Collation -> Either String TextCodec
decodeCodecChunks :: forall (m :: Type -> Type) r. Monad m => TextCodec -> Stream (Of ByteString) m r -> Stream (Of Text) m r
decodeCodec :: TextCodec -> ByteString -> Text
decodeUtf8Lenient :: ByteString -> Text

Documentation

type CodePage = Word16 Source #

A Windows code page number, e.g. 1252. UTF-8 is 65001.

collationLcid :: Collation -> Word32 Source #

The low 20 bits of the collation DWORD (MS-TDS 2.2.5.1.2 Collation Rule Definition).

collationIsUtf8 :: Collation -> Bool Source #

Bit 26 of the collation DWORD is fUTF8.

collationCodePage :: Collation -> Maybe CodePage Source #

The code page of single-byte character data with this collation, if the collation is known.

singleByteDecoder :: CodePage -> Maybe (ByteString -> Text) Source #

The decoder for a supported single-byte code page.

collationTextDecoder :: Collation -> Either String (ByteString -> Either String Text) Source #

Decode character data in the given collation, or explain why it can't be decoded as Text. The outer Either depends only on the collation, so a column can be checked before any of its values are seen.

collationTextEncoder :: Collation -> Either String (Text -> Either String ByteString) Source #

How to encode Text as a collation's charvarchartext bytes, or why hmssql can't: UTF-8 for a _UTF8 collation, else the collation's single-byte code page (singleByteEncoder); the double-byte code pages aren't supported, as for decoding. The encoder fails on a character the code page doesn't have (nothing is replaced by ?, as the server's own conversion would). Used where a value is written for a column of a known collation (Hmssql.BulkCopy).

singleByteEncoder :: CodePage -> Maybe (Text -> Either String ByteString) Source #

Encode Text in a single-byte code page: ASCII as it is, and each other character as the byte whose table entry it is. Nothing for a code page hmssql doesn't have (singleByteTable); the encoder fails on a character the code page doesn't have, naming it.

data TextCodec Source #

How a collation's charvarchartext values are decoded. The codec is selected once per column and consumes a whole value.

collationCodec :: Collation -> Either String TextCodec Source #

The codec of a collation, or why hmssql can't decode it.

decodeCodecChunks :: forall (m :: Type -> Type) r. Monad m => TextCodec -> Stream (Of ByteString) m r -> Stream (Of Text) m r Source #

decodeCodec for a value that arrives in chunks. A chunk can end inside a UTF-8 sequence, so text's incremental decoder holds its bytes back, and whatever is held at the end decodes leniently as a whole value would. In a single-byte code page every chunk stands alone.

decodeCodec :: TextCodec -> ByteString -> Text Source #

Decode a whole value with the codec selected for its column.

decodeUtf8Lenient :: ByteString -> Text Source #

Decode UTF-8 from a _UTF8-collated charvarchartext value, never failing: ill-formed sequences become U+FFFD.

The server stores such values without fully validating them (a varbinary value converted to a _UTF8 varchar keeps most ill-formed sequences), and its own conversion to nvarchar also turns ill-formed sequences into U+FFFD. The well-formed parts decode the same, but the number of U+FFFD replacing an ill-formed sequence may differ from the server's (e.g. ED A0 80, an encoded surrogate, is two U+FFFD in the server's conversion and may be three here). This is text's decodeUtf8Lenient.