Test 1: "This is a test."
This
is
a
test.
Total: 4

Test 2: "This (is) a test."
This
(is)
a
test.
Total: 4

Test 3: "This (is) a test."
This
is
a
test
Total: 4

Test 4: "This (is) a test."
This
(
is
)
a
test
Total: 6

Test 5: "This \"is a\" te\\\"st."
This
is
a
te\"st
Total: 4

Test 6: "This \"is a\" te\\\"st."
This
is a
te\"st.

Total: 4

Test 7: "This \"is 
a\" te\\\"st."
This
is 
a
te\"st.

Total: 4

Reading tokens and binary from string
[TOKEN a]
[TOKEN buffer]
7
-34
[TOKEN and]
[TOKEN tokens]
Reading tokens and binary from file
[TOKEN a]
[TOKEN buffer]
7
-34
[TOKEN and]
[TOKEN tokens]