cuuupid/glm-4v-9b

# cuuupid/glm-4v-9b – Replicate Source: [https://replicate.com/cuuupid/glm-4v-9b](https://replicate.com/cuuupid/glm-4v-9b) This is a vision language model that is highly competitive with GPT\-4’s vision capabilities\. Credit to THUDM, original README in below\. ## Model Introduction GLM\-4\-9B is the open\-source version of the latest generation of pre\-trained models in the GLM\-4 series launched by Zhipu AI\. In the evaluation of data sets in semantics, mathematics, reasoning, code, and knowledge,**GLM\-4\-9B**and its human preference\-aligned version**GLM\-4\-9B\-Chat**have shown superior performance beyond Llama\-3\-8B\. In addition to multi\-round conversations, GLM\-4\-9B\-Chat also has advanced features such as web browsing, code execution, custom tool calls \(Function Call\), and long text reasoning \(supporting up to 128K context\)\. This generation of models has added multi\-language support, supporting 26 languages including Japanese, Korean, and German\. We have also launched the**GLM\-4\-9B\-Chat\-1M**model that supports 1M context length \(about 2 million Chinese characters\) and the multimodal model GLM\-4V\-9B based on GLM\-4\-9B\.**GLM\-4V\-9B**possesses dialogue capabilities in both Chinese and English at a high resolution of 1120\*1120\. In various multimodal evaluations, including comprehensive abilities in Chinese and English, perception & reasoning, text recognition, and chart understanding, GLM\-4V\-9B demonstrates superior performance compared to GPT\-4\-turbo\-2024\-04\-09, Gemini 1\.0 Pro, Qwen\-VL\-Max, and Claude 3 Opus\. ## Model List ModelTypeSeq LengthDownloadOnline DemoGLM\-4\-9BBase8K[🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b)[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b)[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B)/GLM\-4\-9B\-ChatChat128K[🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat)[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat)[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat)[🤖 ModelScope CPU](https://modelscope.cn/studios/dash-infer/GLM-4-Chat-DashInfer-Demo/summary) [🤖 ModelScope vLLM](https://modelscope.cn/studios/ZhipuAI/glm-4-9b-chat-vllm/summary)GLM\-4\-9B\-Chat\-1MChat1M[🤗 Huggingface](https://huggingface.co/THUDM/glm-4-9b-chat-1m)[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat-1m)[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4-9B-Chat-1M)/GLM\-4V\-9BChat8K[🤗 Huggingface](https://huggingface.co/THUDM/glm-4v-9b)[🤖 ModelScope](https://modelscope.cn/models/ZhipuAI/glm-4v-9b)[🟣 WiseModel](https://wisemodel.cn/models/ZhipuAI/GLM-4V-9B)[🤖 ModelScope](https://modelscope.cn/studios/ZhipuAI/glm-4v-9b-Demo/summary)## Projects The following excellent open source repositories have in\-depth support for the GLM\-4\-9B model, and everyone is welcome to expand their learning\. Inference acceleration: - [chatglm\.cpp](https://github.com/li-plus/chatglm.cpp): Real\-time inference on your laptop accelerated by quantization, similar to llama\.cpp\. ## BenchMark ### Typical Tasks ModelAlignBenchMT\-BenchIFEvalMMLUC\-EvalGSM8KMATHHumanEvalNaturalCodeBenchLlama\-3\-8B\-Instruct6\.408\.0068\.5868\.451\.379\.630\.062\.224\.7ChatGLM3\-6B5\.185\.5028\.166\.469\.072\.325\.758\.511\.3GLM\-4\-9B\-Chat7\.018\.3569\.072\.475\.679\.650\.671\.832\.2### Base Model ModelMMLUC\-EvalGPQAGSM8KMATHHumanEvalLlama\-3\-8B66\.651\.2\-45\.8\-33\.5Llama\-3\-8B\-Instruct68\.451\.334\.279\.630\.062\.2ChatGLM3\-6B\-Base61\.469\.026\.872\.325\.758\.5GLM\-4\-9B74\.777\.134\.384\.030\.470\.1> Since`GLM\-4\-9B`adds some math, reasoning, and code\-related instruction data during pre\-training, Llama\-3\-8B\-Instruct is also included in the comparison range\. ### Long Context The[needle\-in\-the\-haystack experiment](https://github.com/LargeWorldModel/LWM/blob/main/scripts/eval_needle.py)was conducted with a context length of 1M, and the results are as follows: ### Multi Language The tests for GLM\-4\-9B\-Chat and Llama\-3\-8B\-Instruct are conducted on six multilingual datasets\. The test results and the corresponding languages selected for each dataset are shown in the table below: DatasetLlama\-3\-8B\-InstructGLM\-4\-9B\-ChatLanguagesM\-MMLU49\.656\.6allFLORES25\.028\.8ru, es, de, fr, it, pt, pl, ja, nl, ar, tr, cs, vi, fa, hu, el, ro, sv, uk, fi, ko, da, bg, noMGSM54\.065\.3zh, en, bn, de, es, fr, ja, ru, sw, te, thXWinograd61\.773\.1zh, en, fr, jp, ru, ptXStoryCloze84\.790\.7zh, en, ar, es, eu, hi, id, my, ru, sw, teXCOPA73\.380\.1zh, et, ht, id, it, qu, sw, ta, th, tr, vi### Function Call Tested on[Berkeley Function Calling Leaderboard](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard)\. ModelOverall Acc\.AST SummaryExec SummaryRelevanceLlama\-3\-8B\-Instruct58\.8859\.2570\.0145\.83gpt\-4\-turbo\-2024\-04\-0981\.2482\.1478\.6188\.75ChatGLM3\-6B57\.8862\.1869\.785\.42GLM\-4\-9B\-Chat81\.0080\.2684\.4087\.92### Multi\-Modal GLM\-4V\-9B is a multimodal language model with visual understanding capabilities\. The evaluation results of its related classic tasks are as follows: **MMBench\-EN\-Test****MMBench\-CN\-Test****SEEDBench\_IMG****MMStar****MMMU****MME****HallusionBench****AI2D****OCRBench****gpt\-4o\-2024\-05\-13**83\.482\.177\.163\.969\.22310\.35584\.6736**gpt\-4\-turbo\-2024\-04\-09**81\.080\.273\.056\.061\.72070\.243\.978\.6656**gpt\-4\-1106\-preview**77\.074\.472\.349\.753\.81771\.546\.575\.9516**InternVL\-Chat\-V1\.5**82\.380\.775\.257\.146\.82189\.647\.480\.6720**LLaVA\-Next\-Yi\-34B**81\.17975\.751\.648\.82050\.234\.878\.9574**Step\-1V**80\.779\.970\.350\.049\.92206\.448\.479\.2625**MiniCPM\-Llama3\-V2\.5**77\.673\.872\.351\.845\.82024\.642\.478\.4725**Qwen\-VL\-Max**77\.675\.772\.749\.5522281\.741\.275\.7684**Gemini 1\.0 Pro**73\.674\.370\.738\.6492148\.945\.772\.9680**Claude 3 Opus**63\.359\.26445\.754\.91586\.837\.870\.6694**GLM\-4V\-9B**81\.179\.476\.858\.747\.22163\.846\.681\.1786## Quick call **For hardware configuration and system requirements, please check[here](https://replicate.com/cuuupid/basic_demo/README_en.md)\.** ### Use the following method to quickly call the GLM\-4\-9B\-Chat language model Use the transformers backend for inference: ``` import torch from transformers import AutoModelForCausalLM, AutoTokenizer device = "cuda" tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-4-9b-chat", trust_remote_code=True) query = "你好" inputs = tokenizer.apply_chat_template([{"role": "user", "content": query}], add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True ) inputs = inputs.to(device) model = AutoModelForCausalLM.from_pretrained( "THUDM/glm-4-9b-chat", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True ).to(device).eval() gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1} with torch.no_grad(): outputs = model.generate(**inputs, **gen_kwargs) outputs = outputs[:, inputs['input_ids'].shape[1]:] print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` Use the vLLM backend for inference: ``` from transformers import AutoTokenizer from vllm import LLM, SamplingParams # GLM-4-9B-Chat # If you encounter OOM, you can try to reduce max_model_len or increase tp_size max_model_len, tp_size = 131072, 1 model_name = "THUDM/glm-4-9b-chat" prompt = [{"role": "user", "content": "你好"}] tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) llm = LLM( model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True, enforce_eager=True, # if you encounter OOM in GLM-4-9B-Chat-1M, you can try to enable the following parameters # enable_chunked_prefill=True, # max_num_batched_tokens=8192 ) stop_token_ids = [151329, 151336, 151338] sampling_params = SamplingParams(temperature=0.95, max_tokens=1024, stop_token_ids=stop_token_ids) inputs = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True) outputs = llm.generate(prompts=inputs, sampling_params=sampling_params) print(outputs[0].outputs[0].text) ``` ### Use the following method to quickly call the GLM\-4V\-9B multimodal model Use the transformers backend for inference: ``` import torch from PIL import Image from transformers import AutoModelForCausalLM, AutoTokenizer device = "cuda" tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-4v-9b", trust_remote_code=True) query = 'display this image' image = Image.open("your image").convert('RGB') inputs = tokenizer.apply_chat_template([{"role": "user", "image": image, "content": query}], add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True) # chat mode inputs = inputs.to(device) model = AutoModelForCausalLM.from_pretrained( "THUDM/glm-4v-9b", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True ).to(device).eval() gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1} with torch.no_grad(): outputs = model.generate(**inputs, **gen_kwargs) outputs = outputs[:, inputs['input_ids'].shape[1]:] print(tokenizer.decode(outputs[0])) ``` Note: GLM\-4V\-9B does not support calling using vLLM method yet\. ## Complete project list If you want to learn more about the GLM\-4\-9B series open source models, this open source repository provides developers with basic GLM\-4\-9B usage and development code through the following content - [basic\_demo](https://replicate.com/cuuupid/basic_demo/README.md): Contains - Interaction code using transformers and vLLM backend - OpenAI API backend interaction code - Batch reasoning code - [composite\_demo](https://replicate.com/cuuupid/composite_demo/README.md): Contains - Fully functional demonstration code for GLM\-4\-9B and GLM\-4V\-9B open source models, including All Tools capabilities, long document interpretation, and multimodal capabilities\. - [fintune\_demo](https://replicate.com/cuuupid/finetune_demo/README.md): Contains - PEFT \(LORA, P\-Tuning\) fine\-tuning code - SFT fine\-tuning code ## Friendly Links - [LLaMA\-Factory](https://github.com/hiyouga/LLaMA-Factory): Efficient open\-source fine\-tuning framework, already supports GLM\-4\-9B\-Chat language model fine\-tuning\. - [SWIFT](https://github.com/modelscope/swift): LLM/VLM training framework from ModelScope, supports GLM4\-9B\-Chat/GLM4v\-9b\-chat fine\-tuning\. - [Xorbits Inference](https://github.com/xorbitsai/inference): Performance\-enhanced and comprehensive global inference framework, easily deploy your own models or import cutting\-edge open source models with one click\. - [self\-llm](https://github.com/datawhalechina/self-llm/tree/master/GLM-4): Datawhale’s self\-llm project, which includes the GLM\-4\-9B open source model cookbook\. ## License - The use of GLM\-4 model weights must follow the[Model License](https://huggingface.co/THUDM/glm-4-9b/blob/main/LICENSE)\. - The code in this open source repository follows the[Apache 2\.0](https://replicate.com/cuuupid/LICENSE)license\. Please strictly follow the open source license\. ## Reference If you find our work helpful, please consider citing the following paper\. ``` @inproceedings{zeng2022glm, title={{GLM-130B:} An Open Bilingual Pre-trained Model}, author={Zeng, Aohan and Liu, Xiao and Du, Zhengxiao and Wang, Zihan and Lai, Hanyu and Ding, Ming and Yang, Zhuoyi and Xu, Yifan and Zheng, Wendi and Xia, Xiao and others}, booktitle={The Eleventh International Conference on Learning Representations, {ICLR} 2023, Kigali, Rwanda, May 1-5, 2023}, year= {2023}, } ``` ``` @inproceedings{du2022glm, title={GLM: General Language Model Pretraining with Autoregressive Blank Infilling}, author={Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie}, booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pages={320--335}, year={2022} } ``` ``` @misc{wang2023cogvlm, title={CogVLM: Visual Expert for Pretrained Language Models}, author={Weihan Wang and Qingsong Lv and Wenmeng Yu and Wenyi Hong and Ji Qi and Yan Wang and Junhui Ji and Zhuoyi Yang and Lei Zhao and Xixuan Song and Jiazheng Xu and Bin Xu and Juanzi Li and Yuxiao Dong and Ming Ding and Jie Tang}, year={2023}, eprint={2311.03079}, archivePrefix={arXiv}, primaryClass={cs.CV} } ```