The adhesive that was used for floor covering applications during that period is, "cutback" and contains asbestos fibers as a binder. There is a simple test that you can do to ascertain the presents of cutback adhesive. Using a sharp blade cut a 2"x2" square through one or both layers of the flooring material and remove the first layer, The exposed adhesive may be black in color. Heat some water to boiling and spill a few drops on the adhesive. Repeat for each layer. If the water drops remain intact, the adhesive is cutback. If the adhesive dissolves and liquifies, you do not have cutback adhesive. Asbestos becomes dangerous and absorbable when it is "friable", IE. airborne. There are a number of citrus based removers which can be safely used.

I found this on https://www.gsa.gov/portal/content/113602 Removing Linoleum Adhesive From FloorsProcedure code:965001SSource:Ohj - Shirley Maxwell & James C. Massey, 9/30/92Division:FinishesSection:Resilient FlooringLast Modified:08/02/2016 REMOVING LINOLEUM ADHESIVE FROM FLOORS PART 1---GENERAL 1.01 SUMMARY This procedure includes guidance on removing the mastic adhesive left behind when linoleum flooring has been removed. Linoleum flooring was very popular from 1863 to 1974. It was used historically as an original floor treatment in some GSA buildings. Linoleum's chief ingredient was linseed oil, pressed from flax seed, which became a tough, elastic material when exposed to air. The process essentially consisted of mixing linseed oil and gum with ground cork or wood flour, and pressing it onto burlap or canvas. It was then glued down to concrete or wood floors using a mastic cement. See "General Project Guidelines" for general project guidelines to be used along with this procedureThese guidelines cover the following sections: Historic Structures Precautions Submittals Quality Assurance Delivery, Storage and Handling Project/Site Conditions Sequencing and Scheduling General Protection (Surface and Surrounding) These guidelines should be reviewed prior to performing this procedure and should be followed along with recommendations from the Regional Historic Preservation Officer (RHPO). PART 2---PRODUCTS 2.01 MATERIALS Clean, potable water Vinegar Citrus degreasing solvent such as "Limo Sol", or approved equal. Dry ice Chemical solvent such as isopropyl alcohol, paint thinner, xylol, lacquer thinner, or paint stripper. 2.02 EQUIPMENT Heat lamp or hot-air gun Putty knife or scraper PART 3---EXECUTION 3.01 ERECTION, INSTALLATION, APPLICATION NOTE: A waterproof glue is usually applied at the edges and seams of the linoleum; the center of the felt layers is usually attached with a water soluble paste. Soak the mastic area in hot water for 20-60 minutes or until soft.\ If this is not effective, try adding vinegar or high-strength citrus degreasing solvent to the water and then soak for 20-60 minutes until soft. If mastic still will not soften, try applying heat to the surface using a lamp or hot-air gun. -OR- Try freezing the mastic with dry ice to break the bond with the substrate. If no other methods are effective in softening the mastic, test a chemical solvent on the area. NOTE: Be sure to test in an inconspicuous area to determine appropriate chemicals and strengths before proceeding with cleaning. Scrape away the softened mastic using a putty knife or garden edger. END OF SECTION interesting................

You're right to be concerned about asbestos exposure. However, the likelihood of asbestos on the surface of the wall is minimal. The "cotton-like fibers" are most likely the paper fibers from the surface of the sheetrock that were roughed up when you removed the panels. There is a possibility that the joint compound could have asbestos in it and you wouldn't want to sand that if it does. If you want to proceed cautiously, contact a local testing lab and collect a couple of samples that can be tested by a qualified, local analytical lab. This would not be very expensive. If there isn't any asbestos, you can proceed with your project. If there is, you wouldn't want to keep working on the wall in a way that could raise dust. Instead, you could patch, but not sand, and then seal the surface with paint. If the surface didn't look as nice as you wanted, you could wallpaper over it with something that had a little texture. Here is a good summary about asbestos in the home and how to involve professionals if you need to. Your state and local environmental health organizations can give you guidance on this, and should be able provide lists of qualified labs and removal specialists. (However, I don't think it will come to that.) Good luck. Here is a link that might be useful: Asbestos - EPA

California homes in 1974 weren't being generally built with ceramic tile floors. It was mostly vinyl tiles or sheet vinyl. The previous owner(s) probably had the tile put in at a later date. By 1978 little to no asbestos was being used in much of anything. Grout or tile having asbestos? No the contractor is being shifty. I think the "contractor " is taking advantage of your inexperience with remodeling and trying to pocket a extra cash on the side. I would let Lowes know about it. They don't need the eventual bad rep guys like this will bring on them. Asbestos fears are overstated anyway. The fear mongering is insane. Removal companies capitalize on that fear. Ship builders and people who have worked around it all their lives get ill. Not a homeowner changing out a floor. Taking up even an old asbestos vinyl floor isn't going to create much dust and you have to breath in the dust for it to to be harmful. If it's a concern wetting the area down will take care of the little bit found in the average home built back in the asbestos days.