You can cut off a chunk and send it off to a lab to test for asbestos content-- obviously, don't do that in an obvious place if you're not immediately planning on replacing it or covering it. If you can lift a baseboard and get material from the edge, then cover the missing section back over with the baseboard, that's often a good location. For the mastic, you can get it wet and scrape up some and send it off for testing. The lab will almost always have specific instructions on safe ways to collect material for testing and how much you need. Similar appearing materials are all over the place on actual asbestos content. Vinyl asbestos tiles tend to be square and have kind of a marbled look. They are often, but not always, 9x9. The vinyl asbestos tiles are generally quite low risk compared to the asbestos-backed sheet vinyl, because the asbestos fibers are bound in a matrix with the vinyl. Unless you pulverize them in removal, you're not going to get airborne fibers taking them out. If you contrast whatever your local recommendations are for VAT removal with sheet vinyl removal, they're usually pretty different-- the main thing people are concerned about with VAT is how you dispose of it. For sheet vinyl, though, they're really worried about how you take it up. Mastics can be pretty difficult to get up. For example, the black mastic was really easy to get up with Bean-E-Doo (whatever they call it now)-- anything else, though, forget it. The remnants of the Bean-E-Doo itself were challenging (lots of soap, water, scrubbing, etc.) However, though the same chemical said it works on old carpet adhesives, it didn't work at all for me. If you have things tested and the tile's asbestos and the mastic's not, the tiles are pretty easy to remove nondestructively with steam (e.g., a steam iron or a wallpaper steamer, or you can rent a big machine that heats the tiles for you). Then you can rent a big commercial scraper to get the mastic up, which is the most efficient. If the mastic does contain asbestos, either it shifts easily with heat or it doesn't, it's water soluble or oil soluble, etc. Use whatever people think is the best method for the specific type of mastic you have. Or paint over the mastic with one of the sealing compounds if the standard removal methods don't work. Since your tiles are loose, though, I'd take them out before putting any new flooring down.

Well, the first thing would be to get it tested. But I agree with bestreviews above, getting it off a wood (or wood byproduct) subfloor is going to be a royal pain in the best of circumstances and a real problem if it does contain asbestos.

It was not hide glue, like that used in taxidermy, although taxidermists would use linoleum paste in their work on occasion. I do not know the formula for linoleum paste, although it is/was a product manufactured from organic materials. There are probably some grain flours and some substances derived from animal hooves incorporated into the product and such. That would be my best guess. Linoleum paste was formulated well before the use of latex chemical compounds and I doubt there would be any petrochemical oil byproducts in the material. Possibly some linseed oil obtained from the seed of the flax plant. Good question though...if I find the exact formulation somewhere, I will post it.

If by "vapor barrier," you mean polyethylene sheeting, not the greatest idea. The moisture drive upwards to an area of lower pressure won't cease, but instead of dissipating, the moisture will condense to water and feed mould growth. Three means to a floating floor: 1) layers of XPS followed by plywood; 2) an airgap membrane such as Delta FL; 3) Dri-Core or equivalent products. Or remove the remnants of asbestos containing materials--if indeed they do contain asbestos**--in a recommended manner or by an abatement company and skim coat away.