自然语言处理NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Generating unconditional samples with GPT-2

自然语言处理NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Generating unconditional samples with GPT-2

• Generating unconditional samples with GPT-2
• Controlling tokenized data
• Generating trained conditional samples
• 星空智能对话机器人系列博客

在医学数据集上训练模型后， 接下来生成无条件样本 python generate_unconditional_samples.py --model_name ‘117M’

#@title Step 11: Generating Unconditional Samples
import os # import after runtime is restarted
os.chdir("/content/gpt-2/src")
!python generate_unconditional_samples.py --model_name '117M'

在服务器命令行输入脚本，运行结果如下

(dl) root@59db211f1693:/Chapter08_2/gpt-2-master/src# 
(dl) root@59db211f1693:/Chapter08_2/gpt-2-master/src# python generate_unconditional_samples.py --model_name '117M'
WARNING: Logging before flag parsing goes to stderr.
W1020 04:38:12.259888 140070790878976 deprecation_wrapper.py:119] From generate_unconditional_samples.py:54: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2021-10-20 04:38:12.261520: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2021-10-20 04:38:12.279426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:02:00.0
2021-10-20 04:38:12.279702: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2021-10-20 04:38:12.280899: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2021-10-20 04:38:12.281981: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2021-10-20 04:38:12.282288: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2021-10-20 04:38:12.283607: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2021-10-20 04:38:12.284625: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2021-10-20 04:38:12.287597: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2021-10-20 04:38:12.289040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2021-10-20 04:38:12.312183: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2021-10-20 04:38:12.345429: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500005000 Hz
2021-10-20 04:38:12.347781: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b06cd2e530 executing computations on platform Host. Devices:
2021-10-20 04:38:12.347831: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): , 
2021-10-20 04:38:12.484360: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b06cd8cef0 executing computations on platform CUDA. Devices:
2021-10-20 04:38:12.484444: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-10-20 04:38:12.486321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:02:00.0
2021-10-20 04:38:12.486443: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2021-10-20 04:38:12.486501: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2021-10-20 04:38:12.486553: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2021-10-20 04:38:12.486604: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2021-10-20 04:38:12.486656: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2021-10-20 04:38:12.486710: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2021-10-20 04:38:12.486777: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2021-10-20 04:38:12.489994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2021-10-20 04:38:12.490099: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2021-10-20 04:38:12.493143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-20 04:38:12.493206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2021-10-20 04:38:12.493234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2021-10-20 04:38:12.496501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10309 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:02:00.0, compute capability: 7.5)
W1020 04:38:12.499568 140070790878976 deprecation_wrapper.py:119] From generate_unconditional_samples.py:56: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

W1020 04:38:12.505186 140070790878976 deprecation_wrapper.py:119] From /Chapter08_2/gpt-2-master/src/sample.py:51: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

W1020 04:38:12.505573 140070790878976 deprecation_wrapper.py:119] From /Chapter08_2/gpt-2-master/src/model.py:148: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W1020 04:38:12.508140 140070790878976 deprecation_wrapper.py:119] From /Chapter08_2/gpt-2-master/src/model.py:152: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

W1020 04:38:14.277628 140070790878976 deprecation.py:323] From /Chapter08_2/gpt-2-master/src/sample.py:64: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W1020 04:38:14.298571 140070790878976 deprecation.py:323] From /Chapter08_2/gpt-2-master/src/sample.py:39: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1020 04:38:14.299457 140070790878976 deprecation.py:323] From /Chapter08_2/gpt-2-master/src/sample.py:67: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.random.categorical` instead.
W1020 04:38:16.669366 140070790878976 deprecation_wrapper.py:119] From generate_unconditional_samples.py:65: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W1020 04:38:16.808486 140070790878976 deprecation.py:323] From /usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2021-10-20 04:38:19.525713: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
======================================== SAMPLE 1 ========================================
When the ground will be completely shaken from left to right by a better surface abutting the ground, it will already be dry, what is less, shall seem left unsatisfactory.

It is still more certain that I have borrowed from a proof;
whether my assertions are sufficiently strong to resist comparison with the rest of
the syllogisms connected with this part of the Critique, even though I am from
the point of view of a sensuous attribute, or whether impressions of this character
form the basis for many other self-evident assertions, in which, while a very
simple proof, the applicability of the rule must have been immediately
distinguished from the principles. We may therefore have at my disposal two
complicated opinions—on the one hand, on the one side of the question whether the
critique is merely formless, and whether it does not assume the identity of a
divine principle and based upon a certain à priori condition, so as to
proceed beyond the limits of experience, and on the other hand, on the
hypothesis-like principle of the deducibility of all appearances after
our own, and the attainment of a transcendental perfection of all
possible experience by means of the experience of a person, I shall, therefore,
be obliged to employ as little auxiliary, as to elicit, the most
right of the apperception of the object in whatever presence that
has been presented to it.

The view of the metaphysic of body is that of the former, which
exists merely to the extent of complex criticism, but is not to be
ferred to above in any other way; but that of the latter takes this
path, and extends it beyond the limits of experience; and we have not
embraced the question out of sight, in the pure teacherical way,
but merely in spirit, if, as numerous sophists now suggest, no
principles of the conception of the object arise from it, and in this
way the categorical system of the axioms of pure perception
is nevertheless a definite field in which to attend altogether, with a man
who is in no certainty certain of his content. But we pursue this path only
unselfishly, and not at the expense of the object which for the
general
reason is of itself less amenable to plurality of ideas, than of great
transcendental demonstrations, before the greatest rational world
could be equipped. But these are not sought after for a purpose
referring only to this faculty, but that they should be employed as some
important means of advancing the speculative interests of reason
attainably into a second, tertiary, and final triumph of the natural
limits of the powers of thought. Now, in the former we lost all hope of
further ascendancy in a certain private sphere without the
maintaining these remarks. But their application still rested, in
the former instance, on the conception of an object other than that of
things—a conception which, as we have stated above, had
always been, of necessity, speculative. If we were to acknowledge on
the part of our fancy an object that had before it nothing of the kind
covered by variety, but which was conceived and prepared immediately and
declared to be accordingly; and in conformity with this conception, having
it in its power to advance the field of practical experience with unforeseen
successions, it seems that we could plainly call the apprehension of
something particularly so, even noble, at the sight of the conceptions
own by those who in a teleological persuasion know that they ought
to contain the conception of reason, the declarations of empirical
practical conviction, and, with equal circumspection, the
descriptive ground of the principle of a transcendental precept.

I now require us to apply the criterion of a maxim of pure reason
to every of the arts of thought. But we must not read about the
representations which the mind has to develop with this capacity of
reception; they may alone suffice for the guiding principles of the will, and
for the final completion of this work, we must refer with this reference
state of mind to the hypothesis that the mind is always an object of
experience—of the power good or evil, with its aswellissimo expression
as a faculty to progress a series of conditions of experience, in
whole, causality—of the power of reason to guide a series of conditions
of experience, in this case alone, over a space of time still smaller than
the series in which we discover itself. These hypotheses admit of
apart from very little empirical experience, and fill quite
manifold grounds for the construction of our investigations in it, and
would not have proceeded so far in any other manner. That, however,
which among self-satisfied
======================================== SAMPLE 2 ========================================
the quantum will not allow
to itself view form, that is to say, of any internal
reality. But
(1) There cannot be a quantum quantity
above the given size, that is to say,
-it must be unbounded in its matter. Consequently, the
quantum cannot be given beyond this limit.

Section II. Of the Fundamental Postulates and the Function of the
Limits of the Pure Understanding

Section I. Of the Different Mathematical Descriptions

Section III. Of the Principles of Conceptions

Section IV. Of the Equations of Conceptions

Section V. Of Conceptions and the Principles of Conceptions

Section VI. Of Conceptions and its æsthetic

Section VII. Conceptions which are Dialectical

Section VIII. Of the Logical Undress

Section IX. Of Conceptions of Transcendental and
Transcendental Objects

Section X. Of Transcendental Objects

Section XI. Of Transcendental Objects

Section XII. Of Transcendental Deities

Section XI. Of Deities of—

Section XIII. Of Completeness, Quality and Empiricism

Section XIIIII. Of Empirical Science and Empiricism

Section XIIIV. Of Judgment and Happiness

Section XIIV. Of Empirical Judgements


*

Hence, indeed the only distinction which I can make in
comparing their conception of power and its relation to nature,
relates to the only synthetical distinction of the two. The
conception of power is, as the reader will understand it,
a conception derived from experience found in experience, and which
persists in the use of the predicate of a hypothetical argument. Without
explanation the reader is bound to infer, that the impossibility of
maxims of a given synthetical THought can be met only when changed by
an intuition or a definition. Secondly, the dialectical nature
of the conception required to refute the necessity
of one conception to apply the other by means of a second requires different
approaches to the former. Here also the rules of the understanding have their place
insensitively and thoroughly in the exercise of this centre. Thus,
the first rule that we are to imitate “one cannot prove one
thing” is no ground of contentment from the exertions of reason; but
upon which, when the contradictory dialectic is suppressed in
question, we must again imitate a rule similar: “All things
must be known”; while on the other hand, that there should appear
in the first place an à priori synthesis which could only be demonstrated
by a supersensible intuition, the second rule that “all
things are possible” is not obtained, but necessarily must be
uncovered in the foregoing path.



Against internal contradictions can be easily opposed to external
opposition. For example, we are urged to uproot contradictions
of universal knowledge by answering, “All objects are communicated by the
principle of universality,” and, in the following case, it is
always imperative, from the inability of the determining least
of our objects—mere ideas—that we should leave a pure respect for
solve the internal contradictions by showed and simple answers.
In a correspondence with the above procedure, the material
momentaplii
would be exactly the same as that proposed in § 2. That is,
this would be just the same as would be produced in conformity with the
undisclosed internal distinction à priori of their nature. For example,
the opposite opposite of contradictions is ever so thus criticized;
that is to say, there is no partiality supposed under the conditions
of these internal dialectical analytical doubts—which is clearly a
possible consequence of the explanation of internal contradictions and
how they are at least hidden under empirical conditions. Now, by
the disposition of the discussion to questions, we may be expected of
something quite different than in the case of internal contradictions, which
cannot be simple contradictions, without being at once understood. It ceases to
be natural to go to these interests, to ask how their
determination is always presented as imperatively objective or as an absolute
account of the relation of a given idea to a possible object. As
the questioner purss forward, he knows only that he has to answer a
question concerning the application of principles borrowed by experience, in
other words, respecting relations which do not necessarily follow
from empirical considerations. If reason conducts us to certain
articular facts in understanding, such as the principles which are
ready to put into practice by the introduction and adducing of sufficient
quantities of materials, without being disposed to employ them, we
cannot be sure that such principles can be proved to be abstracted
from

如果仔细观察输出，我们会注意到以下几点：

• 生成的句子的结构相对可以接受

• 输出的语法不错

• 对于非专业人士来说，输出可能看起来像人类

然而，生成的内容毫无意义,Transformer无法生产与我们训练的医学论文相关的实际内容，获得更好的结果将有助于工作。我们总是可以增加数据集的大小。 但它会包含我们正在寻找的东西吗？我们能找到更多数据的不良相关性吗 ？

设想涉及新冠病毒COVID-19的医疗项目，数据集包含以下内容的句子：

• “新冠病毒-19不是一种危险的病毒，但它与普通流感病毒相似”

• “新冠病毒-19是一种非常危险的病毒”

• “新冠病毒-19不是一种病毒，而是实验室创造的东西”

• “新冠病毒-19肯定不是实验室创造的！”

• “疫苗是危险的！”

• “疫苗是救命稻草！”

• “各国政府没有正确管理这一流行病”

• “政府做了必要的事”

还有很多像这样矛盾的句子。假设您有一个包含数十亿个单词的数据集，但其内容是如此冲突和嘈杂，无论发生什么，你都无法获得可靠的结果 ！这可能意味着数据集必须更小，并且仅限于来自科学论文的内容， 但即便如此，科学家们也常常意见相左。结论是，这将需要大量的艰苦工作和一个坚实的团队来完成可靠的结果。

让我们进一步调查并控制标记化数据。

在本节中，我们将阅读GPT-2模型用其预训练标记器编码的第一个单词。
我们将使用Training_OpenAI_GPT_2_CH08.ipynb 的标记化数据代码，解压文件out.npz，其包含数据集mdset中的医疗论文编码

#@title Additional Tools : Controlling Tokenized Data
#Unzip out.npz
import zipfile
with zipfile.ZipFile('/content/gpt-2/src/out.npz', 'r') as zip_ref:
zip_ref.extractall('/content/gpt-2/src/')

out.npz已解压缩，我们可以读取arr_0.npy，该NumPy数组包含我们正在查找的编码数据集：

#@title Additional Tools: Controlling Tokenized Data
#Unzip out.npz
import zipfile
with zipfile.ZipFile('/Chapter08_2/gpt-2-master/src/out.npz', 'r') as zip_ref:
    zip_ref.extractall('/Chapter08_2/gpt-2-master/src/src/')

out.npz已解压缩，我们可以读取arr_0.npy，该NumPy数组包含我们正在查找的编码数据集：

#Load arr_0.npy which contains encoded dset
import numpy as np
f=np.load('/Chapter08_2/gpt-2-master/src/src/arr_0.npy')
print(f)
print(f.shape)
for i in range(0,10):
    print(f[i])

运行结果如下：

[1212 5644  326 ...   13  198 2682]
(29379,)
1212
5644
326
11
355
716
78
1765
1868
4778

现在，我们将打开encoder.json并将其转换为Python字典

#We first import encoder.json
import json
i=0
with open("/Chapter08_2/gpt-2-master/models/117M/encoder.json", "r") as read_file:
    print("Converting the JSON encoded data into a Python dictionary")
    developer = json.load(read_file) #converts the encoded data into a Python dictionary
    for key, value in developer.items(): #we parse the decoded json data
        i+=1
        if(i>10):
            break;
        print(key, ":", value)

运行结果如下

Converting the JSON encoded data into a Python dictionary
! : 0
" : 1
# : 2
$ : 3
% : 4
& : 5
' : 6
( : 7
) : 8
* : 9

显示编码数据集的前500个标记的key和值

#We will now search for the key and value for each encoded token
    for i in range(0,500):
        for key, value in developer.items():
            if f[i]==value:
                print(key, ":", value)

添加的这些单词，以确保GPT-2预训练标记器能够轻松识别它们，即：

This suggests that

为了确保GPT-2预训练标记器能够轻松识别它们，事实就是如此

This : 1212
Ġsuggests : 5644
Ġthat : 326

我们可以很容易地识别以初始空格开头的初始标记字符（Ġ）。 我们在医学论文中使用以下词语：amoeboid，“amoeboid”是一个生僻词，我们可以看到GPT-2标记器将它分解为几个子词：

Ġam : 716
o : 78
eb : 1765
oid : 1868

让我们跳过空格，看看发生了什么。“Amoeboid”变成了“am”+“o”+“eb”+“oid”，这里不存在未知标记：[unk]，因为使用了字节级BPE策略。

但是，transformer的注意层可能会关联：

• “am”与其他序列关联，如“I am”
• “o”与任何被拆分的序列关联，并包含一个“o”
• “oid”，另一个序列包含“oid”，可能是带有一些算法的“tabloid”

这不是一个好消息。让我们进一步说明这一点：

amoeboid and mesenchymal

以下输出清楚地显示“and”，但至于其余的 标记令人困惑：

人们可能想知道为什么这是一个问题。原因可以归结为一个单词：“一词多义。”如果我们使用word2vec标记器，字典可能不包含生僻词，如“amoeboid”，我们会想出一个未知的标记。如果我们使用字节级BPE，我们将获得总体上更好的结果，因为我们排除了同一单词的较少变体，如“go”和“go”+“ing”，然而，“amoeboid”中的“am”标记将一词多义在低层带入了这个问题，“am”可以是一种前缀x，“am”可以是“I”+“am”中的单词“am”，也可以是类似的子单词，就像“am”+“bush”一样。注意力层可以将“am”作为一个标记与其他“am”，创建不存在的关系。这是NLU中的多义现象的一个核心问题。

我们可以说正在取得进展，但我们需要更加努力地改进NLP。

以上代码的运行结果如下

This : 1212
Ġsuggests : 5644
Ġthat : 326
, : 11
Ġas : 355
Ġam : 716
o : 78
eb : 1765
oid : 1868
Ġcells : 4778
Ġare : 389
Ġless : 1342
Ġcontract : 2775
ile : 576
, : 11
Ġwhile : 981
Ġmes : 18842
ench : 24421
ym : 4948
al : 282
Ċ : 198
cells : 46342
Ġare : 389
Ġmore : 517
Ġcontract : 2775
ile : 576
, : 11
Ġand : 290
Ġthere : 612
Ġmay : 743
Ġbe : 307
Ġa : 257
Ġswitching : 15430
Ġbetween : 1022
Ġam : 716
o : 78
eb : 1765
oid : 1868
Ġand : 290
Ġmes : 18842
ench : 24421
ym : 4948
al : 282
Ċ : 198
m : 76
igration : 4254
, : 11
Ġperhaps : 3737
Ġthere : 612
Ġcan : 460
Ġalso : 635
Ġbe : 307
Ġa : 257
Ġswitching : 15430
Ġbetween : 1022
Ġthe : 262
Ġdominance : 18648
Ġof : 286
Ġchem : 4607
ot : 313
axis : 22704
Ġ( : 357
amo : 18811
eb : 1765
oid : 1868
Ċ : 198
m : 76
igration : 4254
) : 8
Ġand : 290
Ġcontact : 2800
Ġguidance : 11154
Ġ( : 357
mes : 6880
ench : 24421
ym : 4948
al : 282
Ġmigration : 13472
) : 8
Ġ[ : 685
60 : 1899
]. : 4083
ĠOne : 1881
Ġof : 286
Ġthe : 262
Ġmost : 749
Ġinteresting : 3499
Ġ2 : 362
D : 35
Ċ : 198
platform : 24254
s : 82
, : 11
Ġallowing : 5086
Ġto : 284
Ġstudy : 2050
Ġcontact : 2800
Ġguidance : 11154
Ġand : 290
Ġchem : 4607
ot : 313
axis : 22704
, : 11
Ġwas : 373
Ġproposed : 5150
Ġin : 287
Ġ[ : 685
57 : 3553
], : 4357
Ġin : 287
Ġwhich : 543
Ġthe : 262
Ċ : 198
authors : 41617
Ġdemonstrated : 9555
Ġan : 281
Ġadditive : 38298
Ġeffect : 1245
Ġof : 286
Ġchemical : 5931
Ġgrad : 3915
ients : 2334
Ġand : 290
Ġfiber : 13608
Ġalignment : 19114
Ġby : 416
Ġmeasuring : 15964
Ċ : 198
the : 1169
Ġpersistence : 30802
Ġtime : 640
; : 26
Ġthey : 484
Ġalso : 635
Ġobserved : 6515
Ġthat : 326
Ġcells : 4778
Ġwere : 547
Ġdirected : 7924
Ġby : 416
Ġfiber : 13608
Ġalignment : 19114
Ġand : 290
Ġthere : 612
Ġwas : 373
Ċ : 198
no : 3919
Ġeffect : 1245
Ġof : 286
Ġthe : 262
Ġchemical : 5931
Ġgradient : 31312
Ġwhen : 618
Ġfibers : 26742
Ġwere : 547
Ġaligned : 19874
Ġperpendicular : 47190
Ġto : 284
Ġit : 340
. : 13
ĠA : 317
Ġsimilar : 2092
Ġsetting : 4634
Ċ : 198
was : 9776
Ġalso : 635
Ġused : 973
Ġfor : 329
Ġstudying : 11065
Ġthe : 262
Ġdependence : 21403
Ġof : 286
Ġcontact : 2800
Ġguidance : 11154
Ġon : 319
Ġthe : 262
Ġcell : 2685
Ġcycle : 6772
Ġ[ : 685
48 : 2780
]. : 4083
ĠHowever : 2102
, : 11
ĠIn : 554
Ċ : 198
the : 1169
Ġcase : 1339
Ġof : 286
Ġdifferent : 1180
Ġmulti : 5021
- : 12
direction : 37295
al : 282
Ġcues : 25288
, : 11
Ġtotally : 6635
Ġdifferent : 1180
Ġscenarios : 13858
Ġmay : 743
Ġhappen : 1645
, : 11
Ġe : 304
. : 13
g : 70
. : 13
Ġin : 287
Ġ[ : 685
51 : 4349
] : 60
Ġit : 340
Ġis : 318
Ċ : 198
shown : 42579
Ġthat : 326
Ġfor : 329
Ġcontact : 2800
Ġguidance : 11154
Ġand : 290
Ġelect : 1742
rot : 10599
axis : 22704
Ġin : 287
Ġthe : 262
Ġcor : 1162
nea : 39718
, : 11
Ġelect : 1742
rot : 10599
axis : 22704
Ġwins : 7864
Ġwhen : 618
Ġcompeting : 11780
Ċ : 198
with : 4480
Ġthe : 262
Ġdirection : 4571
Ġof : 286
Ġalignment : 19114
Ġof : 286
Ġthe : 262
Ġfibers : 26742
. : 13
Ċ : 198
Multi : 29800
- : 12
cue : 15509
Ġkinetic : 37892
Ġmodel : 2746
Ġwith : 351
Ġnon : 1729
- : 12
local : 12001
Ġsensing : 34244
Ġfor : 329
Ġcell : 2685
Ċ : 198
m : 76
igration : 4254
Ġon : 319
Ġa : 257
Ġfibers : 26742
Ġnetwork : 3127
Ġwith : 351
Ġchem : 4607
ot : 313
axis : 22704
Ċ : 198
Mart : 13143
ina : 1437
ĠCon : 1482
te : 660
ĠâĪ : 18872
Ĺ : 245
ĠNad : 21877
ia : 544
ĠL : 406
oy : 726
ĠâĢ : 564
ł : 254
âĢ : 447
¡ : 94
Ċ : 198
June : 15749
Ġ18 : 1248
, : 11
Ġ2020 : 12131
Ċ : 198
Abstract : 23839
Ċ : 198
C : 34
ells : 19187
Ġperform : 1620
Ġdirected : 7924
Ġmotion : 6268
Ġin : 287
Ġresponse : 2882
Ġto : 284
Ġexternal : 7097
Ġstimuli : 25973
Ġthat : 326
Ġthey : 484
Ġdetect : 4886
Ġby : 416
Ġsensing : 34244
Ċ : 198
the : 1169
Ġenvironment : 2858
Ġwith : 351
Ġtheir : 511
Ġmembrane : 25019
Ġprot : 1237
rus : 14932
ions : 507
. : 13
ĠIn : 554
Ġparticular : 1948
, : 11
Ġseveral : 1811
Ġbiochemical : 47685
Ġand : 290
Ġbi : 3182
ophysical : 41789
Ġcues : 25288
Ġgive : 1577
Ġrise : 4485
Ġto : 284
Ġtactic : 18543
Ġmigration : 13472
Ġin : 287
Ġthe : 262
Ġdirection : 4571
Ġof : 286
Ġtheir : 511
Ġspecific : 2176
Ġtargets : 6670
. : 13
ĠThis : 770
Ġdefines : 15738
Ċ : 198
a : 64
Ġmulti : 5021
- : 12
cue : 15509
Ġenvironment : 2858
Ġin : 287
Ġwhich : 543
Ġcells : 4778
Ġhave : 423
Ġto : 284
Ġsort : 3297
Ġand : 290
Ġcombine : 12082
Ġdifferent : 1180
, : 11
Ġand : 290
Ġpotentially : 6196
Ċ : 198
competitive : 46131
, : 11
Ġstimuli : 25973
. : 13
ĠWe : 775
Ġpropose : 18077
Ġa : 257
Ġnon : 1729
- : 12
local : 12001
Ġkinetic : 37892
Ġmodel : 2746
Ġfor : 329
Ġcell : 2685
Ġmigration : 13472
Ġin : 287
Ġpresence : 4931
Ġof : 286
Ċ : 198
two : 11545
Ġexternal : 7097
Ġfactors : 5087
Ġboth : 1111
Ġinfluencing : 32596
Ġcell : 2685
Ġpolarization : 42704
: : 25
Ġcontact : 2800
Ġguidance : 11154
Ġand : 290
Ġchem : 4607
ot : 313
axis : 22704
. : 13
ĠWe : 775
Ċ : 198
pro : 1676
pose : 3455
Ġtwo : 734
Ġdifferent : 1180
Ġsensing : 34244
Ġstrategies : 10064
Ġand : 290
Ġwe : 356
Ġanalyze : 16602
Ġthe : 262
Ġtwo : 734
Ġresulting : 7186
Ġmodels : 4981
Ġby : 416
Ġrecovering : 20222
Ċ : 198
the : 1169
Ġappropriate : 5035
Ġmacro : 15021
sc : 1416
opic : 16603
Ġlimit : 4179
Ġin : 287
Ġdifferent : 1180
Ġregimes : 25879
, : 11
Ġin : 287
Ġorder : 1502
Ġto : 284
Ġsee : 766
Ġhow : 703
Ġthe : 262
Ġsize : 2546
Ġof : 286
Ġthe : 262
Ġcell : 2685
, : 11
Ċ : 198
with : 4480
Ġrespect : 2461
Ġto : 284
Ġthe : 262
Ġvariation : 12291
Ġof : 286
Ġboth : 1111
Ġexternal : 7097
Ġfields : 7032
, : 11
Ġinfluences : 16717
Ġthe : 262
Ġoverall : 4045
Ġbehavior : 4069
. : 13
ĠMoreover : 10968
, : 11
Ċ : 198
we : 732
Ġintegrate : 19386
Ġnumer : 5470
ically : 1146
Ġthe : 262
Ġkinetic : 37892
Ġtransport : 4839
Ġequation : 16022
Ġin : 287
Ġa : 257
Ġtwo : 734
- : 12
dimensional : 19577
Ġsetting : 4634
Ġin : 287
Ġorder : 1502
Ċ : 198
to : 1462
Ġinvestigate : 9161
Ġqual : 4140
itatively : 48668
Ġvarious : 2972
Ġscenarios : 13858
. : 13
Ċ : 198
Key : 9218
word : 4775
. : 13
ĠKin : 16645
etic : 5139
Ġequations : 27490
, : 11
Ġmult : 1963
isc : 2304
ale : 1000
Ġmodeling : 21128
, : 11
Ġmulti : 5021
- : 12
cue : 15509
, : 11
Ġnon : 1729
- : 12
local : 12001
, : 11
Ġhyd : 7409
rod : 14892
ynamic : 28995
Ġlimit : 4179
, : 11
Ċ : 198

在服务器命令行运行 python interactive_conditional_samples.py --temperature 0.8 --top_k 40 --model_name ‘117M’ --length 50

(dl) root@060ffd5da87e:/Chapter08_2/gpt-2-master/src# python interactive_conditional_samples.py --temperature 0.8 --top_k 40 --model_name '117M' --length 50
WARNING: Logging before flag parsing goes to stderr.
W1020 04:47:45.025585 140575232579328 deprecation_wrapper.py:119] From interactive_conditional_samples.py:57: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2021-10-20 04:47:45.027247: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2021-10-20 04:47:45.044515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:02:00.0
2021-10-20 04:47:45.044825: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2021-10-20 04:47:45.046033: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2021-10-20 04:47:45.047131: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2021-10-20 04:47:45.047448: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2021-10-20 04:47:45.048776: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2021-10-20 04:47:45.049804: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2021-10-20 04:47:45.052806: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2021-10-20 04:47:45.054260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2021-10-20 04:47:45.054613: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2021-10-20 04:47:45.085437: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500005000 Hz
2021-10-20 04:47:45.088032: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560494de83f0 executing computations on platform Host. Devices:
2021-10-20 04:47:45.088134: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): , 
2021-10-20 04:47:45.232670: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560494e46d80 executing computations on platform CUDA. Devices:
2021-10-20 04:47:45.232753: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-10-20 04:47:45.234588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:02:00.0
2021-10-20 04:47:45.234708: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2021-10-20 04:47:45.234764: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2021-10-20 04:47:45.234814: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2021-10-20 04:47:45.234862: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2021-10-20 04:47:45.234911: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2021-10-20 04:47:45.234959: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2021-10-20 04:47:45.235008: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2021-10-20 04:47:45.238164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2021-10-20 04:47:45.238264: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2021-10-20 04:47:45.241313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-20 04:47:45.241363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2021-10-20 04:47:45.241390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2021-10-20 04:47:45.244651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10309 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:02:00.0, compute capability: 7.5)
W1020 04:47:45.247674 140575232579328 deprecation_wrapper.py:119] From interactive_conditional_samples.py:58: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W1020 04:47:45.250178 140575232579328 deprecation_wrapper.py:119] From interactive_conditional_samples.py:60: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

W1020 04:47:45.250591 140575232579328 deprecation_wrapper.py:119] From /Chapter08_2/gpt-2-master/src/sample.py:51: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

W1020 04:47:45.250924 140575232579328 deprecation_wrapper.py:119] From /Chapter08_2/gpt-2-master/src/model.py:148: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W1020 04:47:47.850759 140575232579328 deprecation.py:323] From /Chapter08_2/gpt-2-master/src/sample.py:64: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W1020 04:47:47.867138 140575232579328 deprecation.py:323] From /Chapter08_2/gpt-2-master/src/sample.py:16: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W1020 04:47:47.889066 140575232579328 deprecation.py:323] From /Chapter08_2/gpt-2-master/src/sample.py:67: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.random.categorical` instead.
W1020 04:47:50.205072 140575232579328 deprecation_wrapper.py:119] From interactive_conditional_samples.py:68: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W1020 04:47:50.345160 140575232579328 deprecation.py:323] From /usr/local/miniconda3/envs/dl/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Model prompt >>> 


Model prompt >>> Welcome to the gavin's star space intelligent dialogue robot course
2021-10-20 04:49:23.677542: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
======================================== SAMPLE 1 ========================================
, and the


29. The gavin's star space intelligent dialogue robot course, and the final


30. The gavin's star space intelligent dialogue robot course, 
 31.

       
================================================================================
Model prompt >>> Welcome to the gavin's star space intelligent dialogue robot course
======================================== SAMPLE 1 ========================================


(a) in Space and Time;

(b) in Space and Time: Principles,
(c) in Space and Time: Speculation in Space and Time
(d), and in Space and Time: Speculation in
================================================================================
Model prompt >>> Welcome to the gavin's star space intelligent dialogue robot course
======================================== SAMPLE 1 ========================================


(taken from the gavin's website), or this page for other Gist
searchable content:

http://www.gist.com/p/2d1c6a7e68e1e3
================================================================================
Model prompt >>> 



Model prompt >>> During such processes, cells sense the environment and respond to
external factors that induce a certain direction of motion towards
specific targets (taxis): this results in a persistent migration in a
certain preferential direction. The guidance cues leading to directed
migration may be biochemical or biophysical. Biochemical cues can
be, for example, soluble factors or growth factors that give rise to
chemotaxis, which involves a mono-directional stimulus. Other cues
generating mono-directional stimuli include, for instance, bound
ligands to the substratum that induce haptotaxis, durotaxis, that
involves migration towards regions with an increasing stiffness of
the ECM, electrotaxis, also known as galvanotaxis, that prescribes
a directed motion guided by an electric field or current, or
phototaxis, referring to the movement oriented by a stimulus of light
[34]. important biophysical cues are some of the properties of the
extracellular matrix (ECM), first among all the alignment of collagen
fibers and its stiffness. In particular, the fiber alignment is shown
to stimulate contact guidance [22, 21]. TL;DR:======================================== SAMPLE 1 ========================================
 it as if they were just objects of the senses. But the same is true of matter, which, when it is in motion, finds its way to the surface and is able to connect the matter with the matter of a thing, and when it
================================================================================
Model prompt >>> ======================================== SAMPLE 1 ========================================
 a point
of view, and with them I may know how far we can go in the direction of
motion.

If we consider this question from the standpoint of the reason of
the transcendent, and do not take it that the
================================================================================
Model prompt >>> ======================================== SAMPLE 1 ========================================
 system which is not yet at the end of its state, and consequently, for a given state, a free choice of the system of which this choice is a part. The whole of its internal contingency, therefore, is not to be found in an
================================================================================
Model prompt >>> ======================================== SAMPLE 1 ========================================
 action are, therefore, only the product of a priori principles, and not of an agent determined by principles.

The principle which determines the law of nature must, therefore, be an action of reason. Thus the understanding, by the rule
================================================================================
Model prompt >>> ======================================== SAMPLE 1 ========================================
 be detected by means of a chemical reaction—the internal sense of smell—a phenomenon we call an
antagonism. Antagonism has the meaning of a phenomenon (of sense) which is not merely
phenomenal, but is itself
================================================================================
Model prompt >>> ======================================== SAMPLE 1 ========================================
 the changes in the growth of the organism that make a change in the organism and so on.

But

结果更好，但需要更多的研究。

• NLP星空智能对话机器人系列：第二次星空智能对话机器人Zoom线上演示安排

• NLP星空智能对话机器人系列：StarSpace: Embed All The Things

• NLP星空智能对话机器人系列：Facebook StarSpace框架初体验

• NLP星空智能对话机器人系列：Facebook StarSpace框架案例数据加载

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 多头注意力架构-通过Python实例计算Q, K, V

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 多头注意力架构 Q K V注意力评分

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 多头注意力架构 Concatenation of the output of the heads

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 位置编码（positional_encoding）

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 位置编码案例应用

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 基于BERT模型微调实现句子分类

• NLP星空智能对话机器人系列：第4、5、6、7次星空智能对话机器人Zoom线上演示安排

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 KantaiBERT

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 KantaiBERT ByteLevelBPETokenizer

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 KantaiBERT Initializing model

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 KantaiBERT Exploring the parameters

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 KantaiBERT Initializing the trainer

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 KantaiBERT Language modeling with FillMaskPipeline

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 GLUE CoLA

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 GLUE SST-2

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 GLUE MRPC

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 GLUE Winograd schemas and NER

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Workshop on Machine Translation (WMT)

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 WMT数据处理

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 BLEU smoothing

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Translations with Trax

• NLP星空智能对话机器人系列：第五次星空智能对话机器人线上演示

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 BertViz

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 The Reformer

• NLP星空智能对话机器人系列：重磅信息：10月份一共会安排6次星空对话的机器人线上演示交流

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Pattern-Exploiting Training (PET)

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 The philosophy of Pattern-Exploiting Training (PET)

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 It‘s time to make a decision

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Text completion with GPT-2

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Text completion with GPT-2 step3-5

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Text completion with GPT-2 step 6-8

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Text completion with GPT-2 step 9

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Training a GPT-2 language model

• NLP星空智能对话机器人系列：论文学习 Do Transformers Really Perform Bad for Graph Representation

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Training a GPT-2 language model Steps 2 to 6

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Training a GPT-2 language model Steps 7 to 9

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Training a GPT-2 language model Steps 10

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 T5-large transformer model

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Architecture of the T5 model

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Summarizing documents with T5-large

• NLP星空智能对话机器人系列：业界动态信息分享 自然语言处理中的小样本学习

• NLP星空智能对话机器人系列：业界动态信息分享 Transformer 的性能瓶颈分析与优化

• 自然语言处理NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Matching datasets and tokenizers

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Noisy relationships

• NLP星空智能对话机器人系列：深入理解Transformer自然语言处理 Standard NLP tasks with specific vocabulary